#metabrainz

regarding the highlevel dump, this one won't need to select an element of a json column, so we could just `\copy (select highlevel as ll_row_id, data from highlevel_model where model in (our models))`

ruaok: oh ok. thiking about this, we should probably have a playlist endpoint in LB itself to add an item from just a recording mbid.

(as a future enhancement)

alastairp: i see, nice. lets do two dumps then, one for bpm stuff and one for mood.

great

oh, you'll need highlevel_model.model too, to differentiate between the different rows for a given ll.id

lucifer: we have that. its one of the most basic functions.

highlevel_model.highlevel is the same as ll.id. I'd dump ll.id, ll.mbid separately so that pg doesn't have to do a join

ruaok: oh, i couldn't find it here so thought it doesn't exist. https://listenbrainz.readthedocs.io/en/v-2021-1...

well, how else would we add a track to a playlist?

lucifer: remind me, what exactly was that list of MBIDs?

the outliers in my BPM charts?

the mbids were part of the 165 bpm peak

not a single track I've clicked on is 165 BPM.

the first one - the hi-hat/snare is about 165

yerp, could see that one.

ok, so those cute BPM charts are useless too. 😭

first peak weight is "mean": 0.62 :(

so it seems pretty certain about that. means that we can't always use the weight for this either

[bookbrainz-site] 14MonkeyDo merged pull request #718 (03master…fix#BB-458): Fix[BB-458]: Showing error conflict page on deleting entity twice https://github.com/bookbrainz/bookbrainz-site/p...

what about the second one? Come Meh Way

this is what spotify has for the Cantelowes track ruaok shared earlier. AB has 185 BPM for this.

Is there a way we can sort items by the most confident high level indicator, for example for this one maybe 'loudness', for another track might be 'danceability' ?

yeah, that's why I wanted to consider using bpm_histogram_first_peak_weight, but it seems that there are many cases where it's pretty confident with this value as well

is this an issue with just high bpm recordings or can occur in any type of recordings. if its just high, then we could try querying spotify if a bpm is above a threshold?

but regardless, querying the bpm's spotify has and comparing with AB might be a nice way to detect issues with AB data and possibly identify the issue with the bpm detection alg.

no, sorry

Rescheduled MB search indexes dump to run soon (11:40 UTC) with I/O patch.

ah ok, no worries.

Yep ruaok said the same yesterday :)

The play icon with and without the circle around it aren't the same width

right, as long as you're aware!

Thanks

alastairp: monkey akshaaatt: Freso: invoices please!

it's not december already

I refuse to believe it

On it ruaok

did you do your own research?

I'm trying to, but I can't find any sources which corroborate my belief

but I "feel" that I'm correct

maybe its time to make a site that disavows the fact its Dec already.

itsnotdecemberitsallaliberellie.com ?

Heading to the office, will invoice from there

Thanks for the reminder

[musicbrainz-docker] 14nikosmichas opened pull request #213 (03master…tcp_max_tw_buckets): Limit tcp_max_tw_buckets to avoid connection issues between services https://github.com/metabrainz/musicbrainz-docke...

alastairp: any idea why this would give a 502 ?

Too long a list perhaps?

docs don't specify how many IDs I can include in a call.

maybe it's not visible enough

ah no, i clearly missed that.

but 502?

but good point - we reject listens if they make it to the webserver and there are too many

should be 400, no?

but if you give _so many_ that uwsgi gives up, it returns 502 before it even gets to the AB code

should that be mentioned in the docs?

yeah, I think we can fit that in

this is same as LB-993

LB-993: User feedback endpoint returns 502 when querying too many recordings https://tickets.metabrainz.org/browse/LB-993

yep

we could also increase buffer size for AB like we intend to do for LB

yeah, I don't know what the effect of the buffer size is on the number of workers that we have, I assume it'll increase some memory usage per worker

but its an interesting issue that there's always a point where if it's too large, we get the error from uwsgi not our server

unless you make the buffer size hundres of mb

possibly but we are talking about 4kb here per request, so the overall memory increase shouldn't be much.

buffer-size is limited to 65536 bytes :)

cloudflare has a limit of 32k, other CDNs have a limit of 8k

that query that ruaok showed was only 3.7k, so not sure how it hit the limit

oh, including headers might have put it over

yup

OK, you've convinced me that 8k is fine

out of curiousity, i wonder how would a lookup of say 1000 mbids be done at a time, fat get or post request?

22% match rate on AB on this one example test case. :(

recordings missing from AB?

24% for alastairp, 15 for monkey, 18 for lucifer.

lucifer: yep.

that's a dead end then.

yeah :/

I think AB needs a drastic re-think, alastairp.

what are you looking up?

high level data for tracks that come out of the misssed tracks datasets query.

the last column is mood_aggressive

it suggests that it could work, if we had sufficient coverage in AB.

a shame that we didn't get around to finishing the mbid redirect table task

lucifer: maybe we should just go with the external database for this for now so that we can get a bunch of improvements made

sure, sounds good to me.

maybe I'll have a look at that this week

so something like setting up a cron job that queries all recording_redirect entries since last run and then update recording mbids in AB tables?

no, I'd do it on demand. you query an MBID, it looks up all of the possible redirects, then it returns you data for all submissions for all mbids in the "set"

this way, the results of a query don't change over time

i see. if you want i could take a quick stab at it, put it on ab beta so that ruaok can test it.

lucifer: alastairp : I think you two can stop working on an of this stuff. its just not useful.

lets go back to whatever original tasks we had.

alastairp: do we have any algorithms that we incorporate into a new-AB? Let assume for a second we ditch our AB DB and start over. How would we do that?

can we collect better data with much smaller segment times that would allow us to build better algorithms for better feature detection later?

param: trivial PR for you, if you have a sec: https://github.com/paramsingh/pylistenbrainz/pu...

[bookbrainz-site] 14dependabot[bot] opened pull request #726 (03master…dependabot/npm_and_yarn/normalize-url-4.5.1): chore(deps): bump normalize-url from 4.5.0 to 4.5.1 https://github.com/bookbrainz/bookbrainz-site/p...

ruaok: done! https://github.com/paramsingh/pylistenbrainz/re...

I also added you as a collaborator to the repo for the future

sweet, thanks!

[bookbrainz-site] 14dependabot[bot] closed pull request #724 (03master…dependabot/npm_and_yarn/tar-4.4.19): chore(deps): bump tar from 4.4.13 to 4.4.19 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14dependabot[bot] closed pull request #726 (03master…dependabot/npm_and_yarn/normalize-url-4.5.1): chore(deps): bump normalize-url from 4.5.0 to 4.5.1 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14dependabot[bot] opened pull request #727 (03master…dependabot/npm_and_yarn/postcss-8.4.4): chore(deps): bump postcss from 8.2.4 to 8.4.4 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14dependabot[bot] opened pull request #728 (03master…dependabot/npm_and_yarn/browserslist-4.18.1): chore(deps): bump browserslist from 4.12.0 to 4.18.1 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14dependabot[bot] opened pull request #729 (03master…dependabot/npm_and_yarn/webpack-5.64.4): chore(deps-dev): bump webpack from 5.12.3 to 5.64.4 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14dependabot[bot] closed pull request #717 (03master…dependabot/npm_and_yarn/webpack-5.64.2): chore(deps-dev): bump webpack from 5.12.3 to 5.64.2 https://github.com/bookbrainz/bookbrainz-site/p...

Yeah, that's right !

Go Dependabot, go !

Solving security alerts by removing a security alert mitigation tool. How ironic.

[bookbrainz-site] 14MonkeyDo merged pull request #728 (03master…dependabot/npm_and_yarn/browserslist-4.18.1): chore(deps): bump browserslist from 4.12.0 to 4.18.1 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14MonkeyDo merged pull request #727 (03master…dependabot/npm_and_yarn/postcss-8.4.4): chore(deps): bump postcss from 8.2.4 to 8.4.4 https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] 14MonkeyDo merged pull request #725 (03master…dependabot/npm_and_yarn/path-parse-1.0.7): chore(deps): bump path-parse from 1.0.6 to 1.0.7 https://github.com/bookbrainz/bookbrainz-site/p...

[musicbrainz-server] 14reosarevok opened pull request #2354 (03master…MBS-12114): MBS-12114: Account for the "Disk" alternative spelling to "Disc" https://github.com/metabrainz/musicbrainz-serve...

Threw that tiny change into the milestone ^ :)

Also

Apparently some reports failed to run or something (MBS-12112)

MBS-12112: Several reports are incorrectly empty https://tickets.metabrainz.org/browse/MBS-12112

I tried to log into a musicbrainz-website-prod container and run ./admin/RunReports.pl but I get

root@4bd87d4e8684:/home/musicbrainz/musicbrainz-server# ./admin/RunReports.pl Can't locate List/AllUtils.pm in @INC (you may need to install the List::AllUtils module) (@INC contains: /home/musicbrainz/musicbrainz-server/admin/../lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.30.0 /usr/local/share/perl/5.30.0 /usr/lib/x86_64-linux-gnu/perl5/5.30 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.30 /usr/share/perl/5.30

/usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at ./admin/RunReports.pl line 8. BEGIN failed--compilation aborted at ./admin/RunReports.pl line 8.

yvanzo, bitmap: how do you run reports manually when needed? I know at least bitmap has done that in the past

I mean, I assume the answer is not "run cpanm"

Spotify wrapped has dropped for this year

Ngl it is pretty good

hi reosarevok, I saw more feedback from you on the genres, thanks

np :) Happy to help with whatever comes next

as I said last week, I'll go through the first sheet and hard-code all of our decisions that we weren't able to do automatically. Once that looks good I'll put together a tag submitter

[musicbrainz-server] 14reosarevok opened pull request #2355 (03master…hide-deleted-subscriptions): Don't show deleted users' Subscription tabs to admins https://github.com/metabrainz/musicbrainz-serve...

ruaok: good question. there are new feature extractors in essentia that do more detailed low level features. now that deep learning algorithms seem to give good results (especially for classification), this seems to be the minimum required amount of data

in fact, I was speaking with Dmitry about this a few weeks ago, and he thinks that they're close with a proposed extractor that we could include in AB in order to have better features

are any of them good enough for to think about an AB reboot?

yes, he and I are planning on prototyping a data refresh if not by the end of this year, definitely early next year

are you in the office tomorrow?

I think it's been interesting to see how the scale of AB "breaks" many of the algorithms in essentia - we knew this early on with the machine learning stuff

but the bpm stuff is interesting too. there are a lot of good results, but we just have so much stuff that there is also bad stuff too

office - unsure. laptop is out for repair and I'm not 100% healthy yet

ok, next week then.

but I am rather down on AB right now. I'm questioning the further existence of the project. at very least we need to have a hard look at our short term plans.

right now we're putting on band-aids and performance improvements of something that seems entirely worthless to me.

so, question for next week: What should the MVP for AB so that it can provide some value to its users?

right, we're going to have to think about the value of 100% automated algorithms that we just set and forget

and we need to think about being able to iterate on the algorithms more easily.