#metabrainz

(welcome back from vacay! ^ )

akshaaatt[m]: Will do.

> Moreover, we should add some relevance of the features available on the homepage. This is something which could be general because currently the users will not have a clue of how to use/where to look for such features.

by this do you mean just adding like a block to the front page explaining how listens are used & the features associated with them?

Hi jasondk ! I think for now we can leave out updating the homepage because that would require some thinking as to how we introduce the features. I think updating the docs as of now is the best we can do

Sorry I forgot the dev meeting.(have been focusing on course project recently, our new term is about to start)

Last week I fixed 3 issues of beta: MBS-11931, MBS-11929, MBS-11927 and tried to optimize the UI according to suggestions on the forum

MBS-11931: Beta: Link type option "Discogs" appears under the "get the music" heading https://tickets.metabrainz.org/browse/MBS-11931

MBS-11929: Beta: Untouched URL missing a relationship type may block editing https://tickets.metabrainz.org/browse/MBS-11929

MBS-11927: Beta: URL relationship editor doesn't display invalid relationship type https://tickets.metabrainz.org/browse/MBS-11927

reosarevok, yvanzo : I'm considering removing the numbers and placing an anchor link in duplicate notice, so that users can jump to the item by clicking the link, but how can I use React components in translation strings?

yyoung[m]: something like that? https://github.com/metabrainz/musicbrainz-serve...

you would use exp.l for that

Thanks, I'll have a try

mo’’in’

evening all!

Who admins the Discourse? I wonder if we could turn on the 'mark as unread' function :O

aerozol: that's mostly Freso :)

aerozol: Freso mostly and zas for upgrades, but you can also post to https://community.metabrainz.org/c/site-discuss...

(who will probably see this in a few hours)

thx guys!

[musicbrainz-server] 14reosarevok opened pull request #2254 (03master…MBS-11944): MBS-11944: Fix label on button for place merges https://github.com/metabrainz/musicbrainz-serve...

bitmap: https://tickets.metabrainz.org/browse/MBS-11884 gets even worse, see my comment

MBS-11884: Some old relationship edits have historical type IDs that were re-used by new types

lucifer: https://github.com/metabrainz/sir/pull/111/comm... - no luck finding why that check fails in the first place?

reosarevok: my guess is that the check was relying on some implementation detail of sqlalchemy.

the work one fails similarly, i am trying to apply this workaround to works as well and see if that fixes stuff.

But do we know what it actually wants to check?

If it's not specifically that it is that type

bitmap: left a comment on https://github.com/metabrainz/musicbrainz-serve... because I started thinking about it and got confused about what you meant :)

i think its checking for the column whether we have reached the last column in the "path" and hence should return the value or should we keep on traversing

this is the failing check https://github.com/metabrainz/sir/blob/master/s...

reosarevok: do you remember if any other entity was failing?

lucifer: wait, if all we care about is whether the element is the last in path, splitting on the dots - why are we not just checking what's the last element in path.split('.') ?

Mineo: maybe you can help us here? :)

because of the relationship checks

Ok, hold my hand through this, please :)

So, isinstance(column, InstrumentedAttribute) is meant to be "it's not the last"?

yes right

if the column is not last, then first check if its a value of a relationship to another entity

then based on the type of the relation collect all values in a list.

Just to be clear, when it's not an InstrumentedAttribute, do we know what the type is instead? :)

the one i debugged was a hybrid proxy something.

its probably another internal type, i want to avoid relying on that. i'll see if i can find a public type for it

these are all the public types: https://docs.sqlalchemy.org/en/14/orm/internals...

need to figure out which one those columns fit in

InstrumentedAttribute is there, at least

yes indeed. maybe it wasn't in 1.0 or something changed since then.

So, basically, if it's not the last, it checks for the relationship(s) and collects the values, or if it's not a relationship, it does... the same it would have done if it's not InstrumentedAttribute

also, this is when we use the new `inspect` API, using getattr directly may be perilious...

Would it help to check if column.property exists and is a relationship instead?

And skip the InstrumentedAttribute check entirely?

possibly.

Ain't reverse engineering fun

the issue is we don't have tests so it may break another entity, worse silently.

once we have a way to test the output is correct we can go on simpifying this

So, what I'm hearing is "we should drop this, write proper tests, then come back to it"

(or, if master fails, fix the bare minimum, then do that?)

slightly different, make it work put on test. hope that the bugs break SOLR and we can fix them then

Can't we write actual tests based on a real test DB and then compare the results of the indexing with what we know should happen?

(... assuming we know what should happen)

that's my question!

I mean, that's what we do for most MB tests, anyway :)

yvanzo: do you know how we can get the correct results that should be sent to SOLR? maybe a partial dump of solr's current indices or what sir prod is currently sending to SOLR?

https://github.com/metabrainz/musicbrainz-serve... and whatnot :)

Yeah, having that would be useful :D

that actually reminds we there's search index dump in MB, right?

Do you mean building a sample dataset from Solr cores?

yes sort of that.

lucifer: yes about full dumps: http://ftp.eu.metabrainz.org/pub/musicbrainz/da...

thanks!

ok just checked the instrument one, its solr's internal representation. not what we are looking for currently.

which format are you looking for?

the json sir sends to sold

*solr

we can probably add a patch to have sir dump whatever its sending to solr and run that instance for some time on sir-test.

(run the current master to get this dump)

then we can run the version we are working on currently and compare the results.

you can also use solr admin to check manually what has been entered.

nice that'll help in testing patches quickly. but we'll need the data in a file to write a test suite for sir.

yes

there seems to be two ways at least: either add an option to sir to write to file, or have solr mock writing that to file

lucifer should I continue working on LB-912?

LB-912: Add endpoints and UI to delete feed events https://tickets.metabrainz.org/browse/LB-912

Yeah, I guess we only need one sample, really - then the tests should just be kept updated when intentional changes are made to sir's output but that's about it

riksucks: yes, sure. i'll check and let know the fields that should be used for deleting. in the meantime, you can look at other usages in the codebase.

Seems like all correct with tags from bandcamp

moooin!

alastairp: do you know about songdata.io ? https://songdata.io/track/4cOdK2wGLETKBW3PvgPWq...

Moin and lol😂😂😂

Hello all. Long time no see :) A few suggestions for yvanzo irt fetch-dump.sh 1) It doesn't check to see if the target dump directory exists. It should either error out and tell the user to create them or specify a dump dir or create whatever paths are needed 2) It doesn't seem like DB_DUMP_DIR and SEARCH_DUMP_DIR can be overwritten without changing the script. It would be nice if these

could be configured in .env 3) It would be nice to have options to only fetch files which haven't changed or are missing without having to poke around for wget options to achieve it - since I guess those are the most needed possiblities if a transfer is interrupted, you don't want to start from scartch. 4) if a download fails for some reason after fetching part of the file, try again with

a reget option. 5) consider using rsync (if available). It will likely cut down on bandwidth, though maybe that's been tried and it takes longer with larger files?

Continue submitting tags, but API return 500 error code. What can it mean?

"Never Gonna Give You Up has a lot of energy, which makes it the perfect workout song. Though, it might not give you a strong dance vibe. So, this may not be something that you would want to play at a dance-off." so wrong :D

>look at other usages in the codebase

You mean other tickets?

reosarevok: can you look to see what is causing the 500s that MrClon is seeing?

MrClon: any more info?

riksucks: no, i mean how other db functions work. like deleting a pinning recording, unfollow/follow so on

reosarevok: might be this https://sentry.metabrainz.org/metabrainz/musicb...

Nope, just 500 response code and empty response body. Response headers is {'Cache-Control': 'no-transform', 'X-RateLimit-Reset': '1631001790', 'X-RateLimit-Remaining': '1088', 'Date': 'Tue, 07 Sep 2021 08:03:09 GMT', 'Content-Length': '0', 'X-RateLimit-Limit': '1200', 'Server': 'Plack::Handler::Starlet', 'Connection': 'keep-alive',

'Access-Control-Allow-Origin': '*', 'Keep-Alive': 'timeout=15', 'Content-Type': 'text/plain; charset=utf-8'}

lucifer: I see, on it then

Ok, I'll take a look

MrClon: do you maybe need to escape the & on r&b?

it looks like the xml is invalid.

and MB is giving 500 instead of reporting that

Yeah

I think & needs to be escaped, but this can be improved on our side too

yes, it should be & instead

"Because the ampersand here tells the parser that the following characters are used to represent another character, which in this case would be an ampersand. An ampersand isn't "illegal" in XML-- it just has a special meaning. It means "all of the characters after this until you hit a semicolon should be translated to something else". When you have an ampersand normally, without the descriptive characters and trailing

semicolon, the parser gets confused. "

(https://stackoverflow.com/questions/1328538/how...) :)

Recently i got 502 for invalid requests (invalid XML structure or non XML at all) https://tickets.metabrainz.org/browse/MBS-11940...

MBS-11940: /ws/2/tag return 502 code for invalid requests

I think the difference is that this just breaks the parser :)

So it gives an internal error

maybe zas knows something?

502 is a gateway error

Added https://tickets.metabrainz.org/browse/MBS-11946 for this issue

MBS-11946: An WS tag submission with unescaped & gives an unexplained 500

Unless /media isn't on the local file system but within one of the docker images :)

With escaping script work well, thanks

Yay :)

Check for other xml chars that need escaping just in case, I guess :)

Just do xml.sax.saxutils.escape for tag name

Hi nelgin, nice to see you again :)

3./4. seems to match https://tickets.metabrainz.org/browse/MBVM-64

MBVM-64: Resume fetching partially downloaded dumps after a stop-and-start

bitmap: looking at MBS-960 - starting wondering if it wouldn't be better to "just" implement MBS-5361 here rather than adapting all the methods

MBS-960: Manage tags (rename/remove) https://tickets.metabrainz.org/browse/MBS-960

MBS-5361: *_tag tables should be managed entirely through triggers, not Perl code https://tickets.metabrainz.org/browse/MBS-5361

5. rsync is supported by one US mirror only, and it doesn’t help here because not all of data/fullexport/ has to be downloaded, just the LATEST one.

bitmap: sigh, and I guess that should also apply to _rating_raw vs _meta