#metabrainz

yes, that is a known issue

Now sir is erroring on Solr disconnection…

oh, i wasn't aware of that.

yvanzo: would it make sense to release musicbrainz-docker without the search changes for now?

bitmap: yes, thus the branch mbdb30-legacy-solr7

Successfully imported release! :)

thanks! didn't see you were already on that

can we include https://github.com/metabrainz/musicbrainz-docke... too? 🙏

bitmap: Can you please provide the timestamp for the packet 177010?

yvanzo[m]: 2025-05-19 17:03:33.534972+00

Thank you!

[sir] 14yvanzo merged pull request #164 (03master…fix-id-delete): doc_id for pysolr.delete should be a string https://github.com/metabrainz/sir/pull/164

artist core shows 2x speedup in indexing after patch, running it for all cores now.

[musicbrainz-docker] 14yvanzo merged pull request #299 (03master…eaa-dump): Fetch the event-art-archive dump https://github.com/metabrainz/musicbrainz-docke...

lucifer: running sir?

yes

on trille.

I was doing that too.

Ok, I stopped mine.

oh

Only artist, recording, and url are left.

(plan: if we can smoothly reindex artist & url, then we can start reindexing recording)

sounds good

lucifer: In my experience, reindexing artist may fail after 10-20min, keep me informed.

yvanzo: artist succeeded for me in 10mins.

In mb-docker you mean? Yes.

it used to take 20 mins before today's fix.

I mean in production.

these numbers are for trille.

yes.

ah okay

Oh, ok, an mb-docker on trille.

yes

Again, the main issue is with our main instance, not with mb-docker, even though that is nice to have some improvements.

sorry for the confusion, i am currently not running anything in production.

i wanted to test my fix before deploying it to prod. hence testing it on mb docker to confirm no regressions.

this is the branch, i can build an image and deploy it to prod if you feel its fine

[sir] 14amCap1712 opened pull request #165 (03master…defer-column-property): Improve deferred loading https://github.com/metabrainz/sir/pull/165

I will try it right away.

great

(just patching the container)

I guess that it is partly amending #160.

yes, the gid was accessed by one of the convert function but not added to searchpaths so it would have led to multiple sql queries. it was caught by the raiseloads adding in the new pr.

s/adding/added/

Oh, ok

I wasn't sure if it was needed because a lot of things seemed to be missing from there

I think in the end I added some but didn't even think about track medium

yeah its not very intuitive, that's why added i the raiseload to catch these errors.

[sir] 14amCap1712 closed pull request #154 (03master…raiseload): SEARCH-688 (I): Use raiseload in query loading options https://github.com/metabrainz/sir/pull/154

SEARCH-688: Improve performance and track regressions https://tickets.metabrainz.org/browse/SEARCH-688

[musicbrainz-docker] 14yvanzo opened pull request #306 (03master…mbdb30-solr9): Upgrade MB DB schema version to 30, search, and requirements https://github.com/metabrainz/musicbrainz-docke...

[metabrainz.org] 14fettuccinae opened pull request #506 (03metabrainz-notifications…notification-table): Add required endpoints for notifications (and for digest settings). https://github.com/metabrainz/metabrainz.org/pu...

lucifer: Reindexing artist in production still is disconnected by Solr after some time. Trying again with lower solr.batch_size (100).

yvanzo: any error logs?

It claims to have successfully imported it though!

lucifer: https://gist.github.com/yvanzo/07d4befbe9f0a8a7... (it was called with `reindex --entity-type artist`)

Just a glitch I guess.

bitmap: I opened a PR for each option ^

Currently reindexing url, only recording will be left after that.

that will take ~45-50 mins i think.

i couldn't test recording on trille because it hasn't been updated to latest schema change yet.

I will test that on hip.

thanks

<yvanzo[m]> "bitmap: I opened a PR for each..." <- thanks! there was an existing PR at https://github.com/metabrainz/musicbrainz-docke... too. I think we should delegate these tasks more clearly next time to avoid duplicating our efforts

[musicbrainz-docker] 14mwiencek closed pull request #303 (03master…schema-change-2025-q2): Provisioning for the v-2025-05-19.0-schema-change version https://github.com/metabrainz/musicbrainz-docke...

bitmap: when you are done with MB release, can you please look at https://github.com/metabrainz/docker-server-con... ?

we should probably redeploy sir with that soon, it might help debug the solr connection issues.

yvanzo: before indexing recording in prod, can you please change the logging level to debug? we have some debug logs already in the solr submission code which might help see how many, if any, documents are not being indexed?

lucifer[m]: will do, sorry for the delay

no worries, thanks!

er, that was in reply to the previous message

is sentry.metabrainz.org already pointing to the new instance?

the new instance is at metabrainz-foundation-inc.sentry.io

sentry.metabrainz.org will be retired eventually.

ah, because MBS stopped receiving events recently, so I wondered if they allowed custom domains and it was moved over or something

yeah i noticed that it stopped receiving events and i am puzzled too because it exists and should be working normally.

but then this has happened in the past twice and the only recourse was to soft or hard reset the instance.

lucifer: Can pass the debug flag but it will slow it down even more. (url is still being indexed atm.)

yvanzo: yes fair point. we can update sentry config alternatively.

you can patch the config in the container directly i guess until the PR is deployed.

lucifer: 24min until importing url errored with solr disconnection.

i see

I will try again with lower batch_size and debug flag.

can you point me to the solr 9 nodes? want to see if there is anything useful in solr server logs.

I checked mb-solrcloud-7 logs and found out that someone is looking for an album named “Glitches Errors and Crashes”…

lol

It is the leader for the url collection and did receive a lot of POST /solr/url_… queries.

Started reindexing url with both debug and the new dsn.

zas: should we check haproxy logs preferably for this connection issue?

Which issue?

504s?

SIR got sometimes disconnected from Solr when reindexing stuff.

Last time was at 2025-05-20 19:04:22,399 (running from rakim).

(UTC)

Weird, what is the exact error you get? logs look normal to me on haproxy side on solr 1

pysolr.SolrError: Failed to connect to server at https://solrcloud-privileged.metabrainz.org/sol...: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

bitmap: docker compose release notes are up for review

What does sir in this case? I guess the connection was refused because the node was too busy

submits a batch of documents to the solr node for indexing.

It should retry after a delay, and do few attempts before giving up

yeah i was checking the same whether pysolr the library we use for making these api calls does that or not

since indexes update use quite a lot of resources on solr nodes, connections are maintained during a long time, which increase the risk of reaching limits (250 simultaneous connections per node)

according to https://stats.metabrainz.org/d/ad7f0753-902c-4a... nodes are saturated

that's due to sir for sure

It is running atm

(since 19:21 UTC)

so basically you should decrease the rate of sir queries, in order to let nodes enough resources to process updates

Will do that for the next attempt by decreasing the number of import threads.

Thank you for having looked into it.

the problem is that sir updates are quite heavy, because each update leads to sync between nodes

<yvanzo[m]> "bitmap: docker compose release..." <- after step 5, can we add something like this?

> 6. This is an optional step. If you had previously [built materialized tables](https://github.com/metabrainz/musicbrainz-docker?tab=readme-ov-file#build-materialized-tables), a few of them have to be rebuilt:

> sudo docker-compose exec musicbrainz bash -c 'carton exec -- ./admin/BuildMaterializedTables --database=MAINTENANCE recording_first_release_date artist_release artist_release_group'

looks good to me otherwise, though I haven't tested them

OK, I'm about to test that.

Can you please update the blog post draft accordingly meanwhile?

I just fixed the git tags for MBS and MB docker, removed the Indexed Search section, and added the list of translators from ./po/list_translators v-2025-05-05.0...production

I think it looks good now

lucifer: stopping your mb docker instance on trille, for schema change tests.

sure

url successfully imported in 44min, got disconnected two times only.

[sir] 14amCap1712 opened pull request #166 (03defer-column-property…solr-retry): Add retry-enabled requests session for Solr connections https://github.com/metabrainz/sir/pull/166

yvanzo: i haven't been able to test this yet but if might be useful for recording core ^

bitmap: Got this from running `upgrade-db-schema.sh`:... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

bitmap: Does this issue come from the prebuilt image?

yvanzo: looks like it's running the schema 29 upgrade from last year

is DB_SCHEMA_SEQUENCE up to date?

No

I'm rebuilding a new image.

Started reindexing recording with solr-retry patch.

Running db upgrade in newly prebuilt mb image…