#metabrainz

ruaok: looks like mapping writer is misbehaving. sentry is filled with errors.

ruaok: also, i looked into the failed inc dump. it gives the following error. might be related to the mapping issues.

I have recently joined the Apple Silicon (ARM processors on desktop OS) revolution. Installing Picard right now, let me know if I can provide any assistance, testing, resources, etc. to the project.

I'm thinking about getting an M1 Mac mini, maybe. If there is a community build fleet for Picard or something I might be willing to donate some CPU cycles on a permanent basis with that.

And it will likely take some time until we will have a native ARM build (as universal2 binaries). Currently it even looks like that we would get best support if we upgrade to PyQt6, which in general isn't such a big deal to do, but due to breaking compatibility still means dong a Picard 3 release

And upgrading to Qt6 means end of support for Windows 7, I think also Windows 8 (though nobody probably cares), macOS 10.12 and older Linux distributions

macOS 10.13 also

[musicbrainz-server] 14reosarevok opened pull request #2316 (03master…MBS-12056): MBS-12056: Convert the edit data page to React https://github.com/metabrainz/musicbrainz-serve...

lucifer: moin. rough way to wake up. let me look.

moin!

looking at sentry again, it seems the errors stopped like ~1.5hrs ago.

there is some old container doing some old shit. did you restart anything on gaga today?

no

yeah, because gaga looks ok. but somehow the DB that we use for MB lookups got reverted to the pre-artistst mbid setup.

huh. thats strange

I'll go hunting after I have some coffee.

cool, thanks!

lucifer: I have a feeling that there was an old mapping writer running. now when I want to update the mapping writer, I feel like I have to do a release, when I just want to update one container.

it feels weird to do a release on that.

ruaok: feel free to do a release. but if you don't want to do that, you can use the manual action. both build the image same way just the manual action doesn't create a release on github.

I think I am going to use slightly distinct tag names for releases in that case.

map-v-2021-10-15.0

if they are running special versions.

sure sounds good.

ok. both versions updated with correct keys. can you please mark the issues on sentry resolved?

if they come back, I would like to know.

(due to some reason, sentry didn't group them)

possibly because they came from different threads??

sorry!

possible. but i think also because we are printing errors as string while sentry likes full stack traces.

hah no worries, sentry has a custom search. i'll just use it to merge manually and resolve :D

Not sure it can be improved, but perhaps you know.

if a thread throws and exception, its a like a tree falling in a forest. no one hears it.

but the main thread can capture its exception.

i'll check and if its possible. i usually use exc_info=True but that won't work here.

right

I rather doubt that a reraise from a different thread could work here. so the question is -- how can we raise an exception to make it nicer to sentry?

and yes, the failed inc dump IS related to the mapping issue -- the mapping tables somehow got reverted to an old version of the table/code for a while. HTF??

There's something weird going on with openresty on gateways, not sure what yet, but logs/stats have problems

jeez, everything is cranky this morning. :(

how could the tables get reverted!! that's weird...

eeek, zas.

this appears in error.log, and shouldn't

now, why...

and that's on both kiki & herb

lucifer: I could totally see how is going to the wrong code version could be an issue. but it fixes itself??

did we somehow create a sentinent AI?

more seriously, could this be caused if the MB db switched to the standby temporarily?

the latter is what I was thinking. the former? I think there was stuff growing in my old fridge more sentient than AI....

lol 😆

how do we check if a switch had happened? does it happen only manually or possible automatically?

first I am going to check to ensure that the -writer has the right DB connect strings.

makes sense

because it has access to two MB DBs. prod RO and the sitemaps/json stuff RW

ok, I checked floyd and pink for the mapping scheme -- not there. as expected.

that leaves aretha.

a correct mapping exists on aretha.

lucifer: have you marked all the issues in sentry resolved? I want to look at the error message more carefully

thx

(it'll merge at max 1000 at a time so will take some more time to complete)

ok, yeah, that was was an old mapping writer container running.

v-2021-10-07.0-1-gd70c873

this was the tag.

yeah, exactly.

ok, now that docker services is updated and a unique tag is in place, lets see how it goes.

the problem is that I was running production stuff for a PR that is still in review.

I'll need to think of a better way to manage that in the future. or at least do what I did today.

i see makes sense. i still don't get how the container reverted to an old version and then back to the latest though.

not sure that was it. its a possibility

ah ok.

the problem might've occured during the time the MBID mapping was being rebuilt. but that should be seamless to the users.

at this point we've done all we can. lets keep monitoring.

👍

at least this is ready: https://github.com/metabrainz/listenbrainz-serv...

should i trigger an inc dump and request new stats?

after releasing a new cron container, sure.

sounds good

do you want to retry debugging the stripe stuff?

yes.

what needs to be done next?

cool, i'll update prod and then let's retry the event. maybe it was a temp issue.

meb prod updated. can you retry sending the event now?

done

YAY! 200!

woo it shows up in donations page.

!m lucifer

You're doing good work, lucifer!

!m zas too

You're doing good work, zas too!

i think the fixes zas made yesterday fixed the tls error.

ssllabs or the docker command are now identical for test.meb and meb. earlier they were not.

I really wish we could accept cc payments in EUR. that would be the thing missing to ditch paypal.

yeah indeed :/

[listenbrainz-server] 14amCap1712 merged pull request #1681 (03master…soooper-important-PR): AISOTT: Fix critiquebrainz heading in config.py.sample https://github.com/metabrainz/listenbrainz-serv...

thanks. we do have EUR accounts via Wise and it would be trivial to connect that account.

I found the problem... related to consul-template, and the deployment of serviceregistrator on zappa.... I'll explain once it will be fixed, we are lucky....

stupid consul-template do not support ":" in service names (but consul does)

ruaok: btw i remembered you had fixed the telegraf writer some time ago. can you do it again? it seems to be running on the wrong branch and queries a non existent LB endpoint repeatedly.

lucifer: do you have an error or something I can look at?

switching to herb; to deploy fix; the bug prevented openresty to restart properly, and therefore to switch log files

ruaok: yep, that's what my next question was :D

GET/internal/metrics/stats beta.listenbrainz.org

last i checked the telegraf writer on yehudi is running bu-metrics branch which i feel is almost wrong

switching back to kiki

couldn't gather the courage. lol noice, lucifer

:)

param: <3

I'll restart the metrics writer in a bit.

thanks

that was a messy mess... things should come back to normal now

nice. after a rough start in the morning, thing coming back to normal now :D

lucifer, I think we should also peg metric writer to a tag now. thoughts?

yes, sounds good.

!m zas

You're doing good work, zas!

metric writer deployed, lucifer

ruaok: thanks. telegraf too?

no.

is that still relevant for metric writing?

the code i wrote doesn't rely on that.

no it isn't but an old prototype is running and still hitting LB. we should revert telegraf to a version without metrics.

on clash?

yehudi i thikn

ok, no new commits on master, building and releasing that version.