#metabrainz

yvanzo: sir-prod failed due to consul-template errors

I tried to restart it but now it fails with a Python error

I'll stop it for now, please have a look when you can

I guess the error is because of the high number of messages it has to process

RuntimeError: maximum recursion depth exceeded in cmp

there was an issue with hetzner network around 2AM

2:00->2:24

this was prolly related to maintenance on routers: https://status.hetzner.com/incident/8a03034d-f1... https://status.hetzner.com/incident/42bcd8a7-68...

but the sir issue started yesterday around 19:00 UTC

hi zas: ok, on it.

mooooin

[musicbrainz-server] reosarevok opened pull request #1950 (master…MBS-11419): MBS-11419: Show YouTube Music links on the sidebar https://github.com/metabrainz/musicbrainz-serve...

yvanzo: how did you fix the sir problem?

zas: same as last month: I saved all messages to disk and requeued them gradually.

cannot sir be fixed so it recovers by itself?

done, it's back to normal now.

yes, it should be fixed

I forsee some libertarians suddenly not so keen on free market economics

atj: what has sir done to libertarians :D

argh

my SSH session stalled and I must have pressed up / enter

good start to the day!

Good time-of-day !

tod!

bitmap: I'm pretty sure we talked about the each() issue from https://tickets.metabrainz.org/browse/MBS-11409 at some point, but can't remember more now. Do you remember?

MBS-11409: Recurring warning in server logs for recording's page

atj: I must say this makes for a great everyday waking-up mantra : "I foresee some libertarians suddenly not so keen on free market economics"

morning

[musicbrainz-server] reosarevok opened pull request #1951 (master…MBS-9836): MBS-9836: Guess Case: stop uppercasing "the" in artist names https://github.com/metabrainz/musicbrainz-serve...

yvanzo: around?

I was wondering about why your suggestion in https://tickets.metabrainz.org/browse/MBS-9826 is to start showing hours at 48

MBS-9826: Make edit expiration time show days + hours when it will take more than a day

Mr_Monkey: I say it five times in the mirror every morning. Unfortunately I've been unable to summon Rush Limbaugh so far.

_lucifer: hi, around?

Hah !

[critiquebrainz] alastair merged pull request #345 (master…update-translations): Update translations https://github.com/metabrainz/critiquebrainz/pu...

[critiquebrainz] alastair closed pull request #335 (master…dependabot/npm_and_yarn/bootstrap-4.1.2): [Security] Bump bootstrap from 3.4.1 to 4.1.2 https://github.com/metabrainz/critiquebrainz/pu...

reosarevok: to bring more accuracy for the next day

Do see if my suggestion in the ticket seems good enough maybe :)

reosarevok: the WIP commit is supposed to fix the first commit (of PR #1112) or that is more complicated?

I think it should just fix the first commit to work with the editor data changes

(I wrote this a while ago, but that's what it looked like on review + test)

yvanzo: I wonder why solr nodes have that much load when SIR is working, I guess that's related to Indexes replication, but it seems a lot of load for few changes

mb-solr-5 is writing to disk at a permanent rate of 600mb/s ... other nodes < 30mb/s, any idea?

the graph shows it's reading from disk at 600MB/s?

is the CPU usage user mode usage or non-idle?

yes, sorry s/write/read/

user

some nice graphs there zas, did you create the dashboard?

yes

I do enjoy creating a Grafana dashboard :)

I'll stop this instance, and see how the cluster acts

atj: that's a very good monitoring tool, especially since they added alerts

yeah it's very useful for operational monitoring. I think the alerting functionality is a bit weak though as you can't use templated graphs.

quite strange how the disk read on solr-5 seems to be the only outlier

yes... I'd more than strange ;)

ok stopped this solr instance, let's see if other nodes exhibits same behavior

it doesn't seem another node started to read at this rate, in fact, apart a bit more handled queries (4 nodes instead of 5) nothing changed

I'll start it again

rebooting the node while at it

deploying new LB

[listenbrainz-server] paramsingh opened pull request #1309 (master…param-user-recommendation-event-table): Add new table for user recommendation events https://github.com/metabrainz/listenbrainz-serv...

atj: the template variable support for grafana's alerts is in the work: https://github.com/grafana/grafana/issues/6557#...

ok, the insane read rate issue disappeared after solr restart...

alastairp: hi! i am available now

_lucifer: hi, I just restarted cron

about to do timescale-reader, spotify-writer, and spark

cool, i have logged in to lemmy as well

zas: thanks, I remember looking at that ticket some time ago and seeing 100 odd "+1" comments!

alastairp: i was wondering why the registrator did not restart in case of lb containers like it happened in case of CB

_lucifer: because I'm starting services manually

makes sense

`source scripts/services.sh` and then manually run `start_listenbrainz_api_compat 'prod' 'v-2021-03-01.1'`

in the meanwhile, should i build beta and test images so that we can update them as well in everything goes right in prod

sure, go ahead with building beta

test isn't too important - we only use that for testing major changes

👍

restarting web

do labs-api soon, plz. the new web relies on labs-api.

LB back up

yes, I did both at the same time

👍

all containers now running 2021-03-01.1

!m alastairp

You're doing good work, alastairp!

quiet, nagios

crap, the labs api throws errors. I'm on it.

I see it. misconfigured typesense url?

likely. in consul temp?

the api key seems to be wrong

it throws a 404. would it do that for a wrong api key?

ruaok: do you want me to roll back to old web and labs-api for now, or do you think that we can fix it shortly?

I can fix shortly.

fair point!

alastairp: beta's pushed

its a cannot connect problem. so URL construction goes wrong somehow.

ruaok: is it possible that the mapping isn't there?

curl http://10.2.2.31:8108/collections/ -H "Content-Type: application/json" -H "X-TYPESENSE-API-KEY: ---the---key---"

[{"created_at":1614658498,"default_sorting_field":"score","fields":[{"facet":false,"name":"combined","optional":false,"type":"string"},{"facet":false,"name":"score","optional":false,"type":"int32"}],"name":"mbid_mapping_20210302_041458","num_documents":15283426,"num_memory_shards":4}]

seems that it didn't rename it from mbid_mapping_20210302_041458 to mbid_mapping

there is a concept of collection aliases. there is an alias from mbid_mapping to mbid_mapping_20210302_041458. Changing this alias allows the atomic rotating of indexes.

this problem is a connection problem first and foremost. the config info that LB labs has is correct, but the container cannot reach typesense.

the alias is called mbid_mapping_latest, but the config at https://github.com/metabrainz/listenbrainz-serv... is looking for mbid_mapping

uhm, for some reason the connection now works.

huh.

ok, agreed, that makes sense.

I'll just commit the fix directly to master. should I push v-2021-03-02.1 ?

just do the commit, _lucifer can do the release + push :)

commit pushed.

thanks _lucifer

it'll be 02.0, I released yesterday's build

on it

_lucifer: git tag v-2021-03-02.1, merge master to production, make github release, build+push image

uh

v-2021-03-02.0

I'm really not a fan of magic python HTTP API wrappers that hide the details of the request that it's making. all such clients should have a straightforward way of debugging the details of the requests

"magic python HTTP API wrappers" like what?

like the typesense client library

I call client.collections['foo'].documents.search, but what http request is that actually making

I ran into exactly the same problem with another client yesterday. Once I could work out what query it was trying to make, the problem was obvious, but it took ages to get to that point

I HATE that API bit.

terrible for testing as well.

[listenbrainz-server] release v-2021-03-02.0 has been published by github-actions[bot]: https://github.com/metabrainz/listenbrainz-serv...

OK. the only remaining python apps on the old version of consul-template are sir and wikidata bot

!m _lucifer