#metabrainz

yvanzo, bitmap: woops, I got BG3 and completely forgot we were meant to meet yesterday D:

aerozol: good evening, can do today if around, or tomorrow.

yvanzo: evening! Something came up for the next couple of hours, but I’ll see if you’re still free (and I’m still awake) afterwards if that’s okay

okay, just ping me if around later on, or we will have another try another day :)

mayhem, bitmap: we already had such load issues with queries using OFFSET in the past (but I don't think it was YandexBot). Perhaps we should target such queries to one read-only server, instead of hitting main rw db. We could do that for all identified indexing bots btw.

yvanzo: it seems https://app.circleci.com/pipelines/github/metab... (merge to master) has a perlcritic issue, did you look into it yet or should I?

reosarevok: might be my own mess, will check, thanks

It probably is just having an extra space somewhere or whatever :)

IIRC, last time it was some lidarr addon that was doing WS queries for recordings with large offsets

reosarevok: I played BG3 much before release (alpha/beta, at least 3 times), and definitively will soon again ;) Another game I had a lot of fun with recently is Wartales

yvanzo:

yes, I saw

It's a bashism from my latest merged PR

It's working Perl, it's just confusing :)

I think the Perl::English equivalent is $OS_ERROR?

In bash it is the return code of the latest commad, in perl a string is expected.

that's $? in bash

$! is PID of last executed background task

right

never seen a use of $! in bash

zas: I'm trying a gnome bard with a drum, it seems fun :D

reosarevok: do you have a copy of PBP?

I do not

those errors always annoyed me as I didn't have a copy. most linters actually have a website with an index of each rule and what it means

but it is a 2006 version and page 79 doesn't match.

yeah. The perlcritic rule modules usually explain it too

But it would be easier of the linter errors linked to them tbh

reosarevok, atj: Apparently $! contains the system error in Perl: https://perldoc.perl.org/perlvar#Error-Variables but it doesn’t follow the PBP somehow.

We have a rule that enforces using more understandable names for these variables, just rename to $OS_ERROR and make sure the file does use English

The rest of our Perl code mostly avoid using $! though, probably for a good reason.

I did.

It’s just not a bash-ism.

Gotcha :)

[musicbrainz-server] 14yvanzo opened pull request #3007 (03master…fix-dollar-bang): Fix CI tests: Replace Perl short `$!` with long `$OS_ERROR` https://github.com/metabrainz/musicbrainz-serve...

reosarevok: The cause of it was I did not rebase the PR on `master`.

So the CI tests were passing for the PR despite the Perl linter config has changed in `master`.

It’s usually fine for recent PRs but this PR has been opened for 20 months.

Yeah, makes sense :)

I try to rebase on master for all old PRs, but it's easy to forget if there's no conflicts

No big deal

Yes, this code is not covered by CI tests but for linting.

[listenbrainz-server] 14amCap1712 merged pull request #2538 (03master…standalone-recs-fix): Handle standalone recordings in spotify id lookups https://github.com/metabrainz/listenbrainz-serv...

yvanzo: I'm still up, but not feeling awake enough to absorb information. I have band practice tomorrow night but will see if I get back early

Really appreciate the help!

no problem, can try next week

[metabrainz.org] 14amCap1712 merged pull request #420 (03master…react): Add react https://github.com/metabrainz/metabrainz.org/pu...

[metabrainz.org] 14dependabot[bot] opened pull request #422 (03master…dependabot/npm_and_yarn/semver-5.7.2): Bump semver from 5.7.1 to 5.7.2 https://github.com/metabrainz/metabrainz.org/pu...

mayhem: Not the issue I thought it was, but the problem was indeed between my chair and my keyboard.

[musicbrainz-server] 14yvanzo merged pull request #3007 (03master…fix-dollar-bang): Fix CI tests: Replace Perl short `$!` with long `$OS_ERROR` https://github.com/metabrainz/musicbrainz-serve...

[metabrainz.org] 14amCap1712 merged pull request #375 (03master…subscription): Subscriptions using Stripe https://github.com/metabrainz/metabrainz.org/pu...

mayhem: tested it locally and it worked fine. i think its ready to be put into prod.

lucifer: go for it!

[metabrainz.org] 14amCap1712 opened pull request #423 (03master…misc-fixes): Misc Improvements https://github.com/metabrainz/metabrainz.org/pu...

mayhem: will do it together with this ^

[listenbrainz-server] 14MonkeyDo opened pull request #2540 (03master…lb-radio-fixes): LB Radio beta: Fix front-end issues and styling https://github.com/metabrainz/listenbrainz-serv...

mayhem: ^

lol just had an insane idea: how about a n app that uses ai or some other bullshit wher yo ucna take images of instruments and thne it will identify for you the instrument and show the info we have of it in app

mayhem: for top releases of artist, what all data do you need?

[listenbrainz-server] 14amCap1712 merged pull request #2539 (03master…popularity-api): Add API endpoint to retrieve artist's popular recordings https://github.com/metabrainz/listenbrainz-serv...

mayhem: Fixed a css linting issue, tests are re-running

lucifer: We should aim to finish up LB#1800 when we have a minute. I'm happy to merge the functional back-end parts and work on the UI in a separate PR if it makes it easier

Import from spotify extended streaming history: https://github.com/metabrainz/listenbrainz-serv...

monkey: we should decide where we want the bulk of processing to happen for that.

lucifer: from a technical perspective?

[listenbrainz-server] 14MonkeyDo merged pull request #2527 (03master…dependabot/npm_and_yarn/word-wrap-1.2.4): Bump word-wrap from 1.2.3 to 1.2.4 https://github.com/metabrainz/listenbrainz-serv...

mayhem: yes

do we want to run it like LastFMImporter on the client side or on server side?

I would say server side. this way the user has less chance of mucking it up, but we have more work to do.

On server side means uploading thos big json files, right?

yes

we need to add a way to store those temporarily

rabbitmq queue for the json files and an importer that consumes those.

also these files have IP addresses

And maybe even to unzip the zipped file to prevent some errors?

not much we can do about that. promise users we won't look at them.

Hmm.

Is that valid GDPR-wise?

yeah if we are going to do it backend side that's the best we can do.

unzipping -- the is one zip program at spotify zipping them. I doubt that will be a problem.

GDPR wise if we discard the private info, yes. we should just make it clear that we are doing that.

Do we want to let the user upload a zip file or just the particular json file?

otherwise we would need to get permission to 'process' the data.

the zip file has many json file s that are irrelevant to us

zip. one less step for the user, less bandwidth used, less rabbitmq space.

Files — plural. That's why I mention the zip, it's less error-prone to upload all of it in one go

yerp

Here's what mine looks like: https://usercontent.irccloud-cdn.com/file/osuOr...

Forget the formatted directory, that was me trying something

i see so this will need more care.

users can upload malicious zip files to ddos us

we did this not because we thought...

lucifer: I dont see that as a great problem.

out of curiosity, why not do it client-side and avoid those thorny issues?

limit file size. be very strict in what gets accpted.

zip bombs

client side could mean network connectivity issues, users browsing away from the page, browser issues, etc.

Mmm, but the process can be safely repeated, right?

IMO that might be better considering the issues this creates

I suppose one can create a zip bomb with other compression formats as well.

yes probably

one way to mitigate zip bombs would be to use a sub-process with CPU and memory restrictions.

or to unzip into ram and if you generate more than X MB of data, killit.

yeah makes sense. just pointing out there are issues with doing it server side.

mayhem: how about a small executable to upload it from client side?

there are other benefits of server side too. we can control the rate of which things get imported.

executable? like a binary?

yes

HELL NO.

do it client side then.

the nightmare of producing 3 binaries is a nightmare.

if you really don't like server side, then do client side.

I don't really care that much.

okay let's do it client side for now then. sending files over rabbitmq is also highly discouraged.

i.e. unzip the file, upload all the files ending with .json

Hm, one thing we might need lucifer is a way for users to set a date until when to import, to avoid duplicates. What do you think? Maybe a begin and end date, optional?

monkey: yes makes sense.

i am looking at js libraries to see if extracting in browser is feasible.

Oh, extracting the zip? interesting. never thought of doing that

in the browser i mena

[listenbrainz-server] 14MonkeyDo merged pull request #2540 (03master…lb-radio-fixes): LB Radio beta: Fix front-end issues and styling https://github.com/metabrainz/listenbrainz-serv...

anything else that should get merged for a release, lucifer ?

mayhem: nope, seems good to me

ok, i"ll release.

monkey: yeah, if we can do that then let's do it client side otherwise server side. i think most people wouldn't go to the pain of submitting each file manually after extraction.

Fair

[listenbrainz-server] release 03v-2023-08-10.0 has been published by 14github-actions[bot]: https://github.com/metabrainz/listenbrainz-serv...