#metabrainz

(because that PR changed a _lot_ of lines)

yeah makes, its now just adding a handler in initial conf and be done with it.

*makes sense

i was thinking to remove pytest deps from reuirements_spark.txt by add another step to build test image in Dockerfile.spark.thoughts?

yeah, no problem. though can we reuse requirements_dev to install them?

no actually we have them in requirements_spark.txt as well.

ah yes right we can

yeah, great. I just didn't want to hard-code dependencies in Dockerfile, and I didn't want _another_ requirements file

yvanzo: around?

alastairp: what do you think we should do about pyspark dep? adding to requirements_development.txt seems wrong. if we keep in requirements_spark.txt it'll be for sake of tests only.

ah, interesting question

also, i saw you added `sentry_sdk[pyspark]` instead of just `sentry_sdk` any particular reason?

I think it makes sense to keep it in requirements_spark.txt, it feels like the better place for it

mmm, good point. sentry_sdk[pyspark] ensures that pyspark is also installed (it makes it a dependency)

i don't think we want that as pyspark should be provided externally by spark

and we uninstall it just after anyways

but if the spark setup provides pyspark without us needing to add it, perhaps just sentry_sdk is OK

why do we uninstall it?

I don't know how the spark/pyspark thing works

how does spark provide pyspark? does it put it in sys.path? is it before or after the rest of our dependencies?

because we pack the deps in venv in a zip and send it to all spark workers

reosarevok: yup

yvanzo: see my comments on the PR :)

i am not familiar with how it works under the hood to supply pyspark.

neither am I

sorry, I don't have a lot of time today. I'll be at the office tomorrow and would be happy to talk through it with you

sure, thanks!

reosarevok: thanks, replied!

Mr_Monkey: ping

Hai !

I saw you added some mockups here

which one should we go ahead with?

Which one do you think looks best and clearest?

I'm not a fan of option #1, a bit simple.

the last one super nice :D

Let's go with that then :)

second one is also nice

We can always revisit the design after an initial implementation

sure, do you have this implemented or is it to be done?

No, I don't have any code for it I'm afraid

That day I got lazy and did the page designs directly in an open browser page…

no worries, i'll modify the spotify page for now and try to put the youtube player and oauth up today for testing.

ruaok: i think that the redirect uris in syswiki might not work. for my local setup, i had to add the complete url including the path.

regarding the youtube api key

update syswiki, plz

we'll have to update in google api console first.

the console will give a new configuration file after that.

ok, logged in. what changes need to be made?

http://localhost/profile/music-services/youtube... this is what i am using for my local api key

if we are fine with the url structure then, adding `/profile/music-services/youtube/callback/` at the end of all urls should be enough

including trailing slash?

yes

perfect, thanks!

we'll need to add some redirect uris for spotify as well to test on beta. but not needed urgently that can be done tomorrow.

Cross-posting this from #musicbrainz

Hello everyone! I'm building a website that will allow guitarists to upload Guitar Pro files–as part of this I want to use MusicBrainz to lookup additional metadata about the artist such as their canonical name, aliases, genres, etc.

I've looked over the schema for the DB but I'm still a bit unsure how to traverse it to get the information I

want. My thinking was that I would search for the "Work" by name and then get to the Artist from that. Using "Toxicity" as an example however you can see that the relationships do not include "System of a Down" 

https://beta.musicbrainz.org/work/ab2a78d4-b0ce-391e-bee5-680d69807bf7

It looks as though I would have to inspect the recordings

to somehow derive the "canonical" recording/artist but I'm unsure if my assumptions are correct or if there is another way to go about this.

Any help in regards to this would be greatly appreciated!

_lucifer: syswiki updated.

thanks!

reosarevok: can you please help kyledecot?

Mr_Monkey: in the mockups I see we do not show import details there. that's expected.

s/expected/intended?

How can I determine if live indexing is turned on with the musicbrainz vm?

and working

if you followed https://github.com/metabrainz/musicbrainz-docke... live-indexing should probably be working, otherwise there

is a bug

I got the MBID matcher running live against incoming listens:

listens: exact 1785 high 20 med 154 low 55 no 873 err 0

68% match rate. :)

no trouble keeping up with incoming listens.

kyledecot: hi! Give me a moment if that's ok? I'm making dinner :)

on thursday I'll add support for working through old listens.

Well, my system spent from 2:17am to 4:06pm UTC loading the indexes and it's still not finding any results.

kyledecot: oh, I see you're getting help in #musicbrainz, will check for a mo

yvanzo: can you help nelgin ?

kyledecot: oh, no, I'm being dumb. Sorry. Ok, I'll quickly look, food is in the oven anyway

So

Yes, you can't trivially get from the work to a "canonical artist", because we don't really have those

(for example, many works don't really have those, such as classical, folk and even some jazz music)

For what you want, the most common case is going to be "find a recording, and pick the artist". But do you have any artist name to begin with?

Because if all you have is, say, the word "Dumb", there's no good way to know if you should be picking Nirvana or Garbage (or someone else)

reosarevok Yeah I have the artist name to begin with (it's encoded into the Guitar Pro file). I thought I should start at the "work" to also get the canonical title (in case they submit something like "toXiciTY" or something.

Well, the problem is sometimes we won't even have works, especially for popular music

Those are not added automatically, they're added by users mostly when they need to either link covers or add a composer / lyricist

(not guitar music but): https://musicbrainz.org/ws/2/recording?query=ar...

Oh I see–so would it be safer to just try and look up the artist directly and then attempt to get the work separately (basically split this into two problems)?

I would suggest trying to find a recording using title + artist

And then you can check whether the recording is linked to any works if you want info from them :)

(if it is, you could use the work title as the standard title, while if not, you could just default to the recording title, which might already be an improvement - for example, a search for toXiciTY would probably give you a recording named Toxicity anyway :) )

reosarevok that all makes sense–my main blocker was just knowing which resource / entity to start with. I'll probably have additional questions as I continue to develop this but you've done a great job in giving me a direction to head in. Thanks for the quick response / building such an awesome / open platform!

kyledecot: neat! Don't hesitate to come back and ask more

_lucifer: missing import details is not intended, no. We'll have to find some space for it

Mr_Monkey: should we put it on another page?

I think it makes sense to keep it with each music service in that page. Maybe an "Import details" accordion hidden by default?

yeah that could work

nelgin: Solr works asynchronously. It takes extra time for search indexes to be readily available.

I’m working on improving stuff around BASE_FTP_URL variable you mentioned yesterday.

When live indexing is working, reindex messages are queued. So you can check either logs of 'mq' or 'search' services.

'indexer' logs can also be useful: it indicates when reindex messages are processed with timestamp.

sudo docker-compose logs --tail 10 --timestamps indexer

yvanzo ah, did I screw things up?

indexer_1 | 2021-04-27T16:25:02.607307633Z This probably means the server terminated abnormally

indexer_1 | 2021-04-27T16:25:02.607310064Z before or while processing the request.

indexer_1 | 2021-04-27T16:25:02.607315104Z [SQL: 'SELECT recording_1.id AS recording_1_id \nFROM musicbrainz.recording AS recording_1 JOIN musicbrainz.artist_credit ON musicbrainz.artist_credit.id = recording_1.artist_credit \nWHERE musicbrainz.artist_credit.id = %(id_1)s'] [parameters: {'id_1': 15856}]

Hm, if I go back further

indexer_1 | 2021-04-27T16:24:40.431919098Z 2021-04-27 16:24:40,431: Requeuing 100 pending messages.

indexer_1 | 2021-04-27T16:24:40.441542857Z 2021-04-27 16:24:40,441: 100 messages requeued.

indexer_1 | 2021-04-27T16:24:57.994325142Z 2021-04-27 16:24:57,993: Error encountered while processing messages: Post to Solr failed. Requeueing all pending messages for retry.

I'm going to pastebin this entire log - tehre's all sorts of stuff in here.

Please pastebin the ouput of this command too: sudo docker-compose exec mq rabbitmqadmin -u sir -p sir -V /search-index-rebuilder list queues

delete, failed, index, and retry all 0

Too big to pastebin so it's here on my server https://wibble.sysadmininc.com/log.txt

I tried to download and import the indexes twice but had no job per previous comments so ended up just rebuilding them.

Never seen "this IndexWriter is closed" message before.

It seems it might be Solr ran out of resources.

Next time your replication cron task is running, there should be new messages in 'indexer' logs again.

I can run it now if you like, it wont be for another 14 hours otherwise

Well, I should say I can run it manually

Okay

Ok, replication running

search.index | 1722

indexer_1 | 2021-04-27T17:37:50.642120375Z 2021-04-27 17:37:50,641: Successfully processed 100 messages

So it looks like its working

Though I just got this in my replication log

WARNING: amqp could not commit tx mode on broker 1, reply_type=2, library_errno=4

:|

Messages are queued and processed, so it’s working indeed.

The warning is about https://rabbitmq-c.docsforge.com/master/api/amq... but that requires more investigation.

It seems to happen a few seconds after I run sudo docker-compose exec mq rabbitmqadmin -u sir -p sir -V /search-index-rebuilder list queues

I just tried it again and got the same thing.

Ok, just ignore this warning then.