first, a note that this whole block will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-an… is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/hoFHDgPhaqLiVUdjSzFWClyL>)
2025-05-20 14054, 2025
julian45[m]
hmm, i think i see part of the problem here
2025-05-20 14029, 2025
julian45[m]
in that this task, even if it could successfully run, is not pulling in the latest work in mbssss; it's pulling in an old commit
2025-05-20 14014, 2025
julian45[m]
s/mbssss/mbsssss/
2025-05-20 14042, 2025
julian45[m]
if we look [one level above the local directory we want to pull config directories from], we'll see that its copy of the mbssss dir is pulling in [mbsssss @ e2e630a](https://github.com/metabrainz/mbsssss/tr…, which is nearly a year old... the submodule has not been updated in metabrainz-ansible for some time
so, absent my concern about the copy_links and links params potentially stepping on each other, that task should run at this time, though it will likely not produce desired results due to the outdated submodule
2025-05-20 14025, 2025
julian45[m]
* so, absent my concern about the copy_links and links params potentially stepping on each other, that task seems like it should run at this time, though it will likely not produce desired results due to the outdated submodule
2025-05-20 14040, 2025
julian45[m]
* so, absent my concern about the copy_links and links param values potentially stepping on each other, that task seems like it should run at this time, though it will likely not produce desired results due to the outdated submodule
2025-05-20 14043, 2025
julian45[m]
bitmap[m]: fwiw, yvanzo had opened a PR to update all of the submodules in metabrainz-ansible ~1 month ago, but that still wouldn't have caught the work done in the past few weeks
2025-05-20 14015, 2025
julian45[m]
* first, a note that this whole block will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-an… is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/BUYPoWljJYzhSmetWtXcYdLZ>)
2025-05-20 14041, 2025
bitmap[m]
yeah, looks like we need to update the .jar file (solr_shared_libraries) too
2025-05-20 14050, 2025
julian45[m]
oh yes, and have solr_shared_libraries point to metabrainz/mb-solr rather than yvanzo's fork
2025-05-20 14015, 2025
bitmap[m]
I guess the intent of his previous PR was to only update the roles/ anyway (but conveniently there were no mbsssss changes at the time, that's the only non-role submodule)
2025-05-20 14025, 2025
julian45[m]
yeah, looks like it
2025-05-20 14037, 2025
julian45[m]
oh, and solr_version should be 9.7.0, right?
2025-05-20 14028, 2025
julian45[m]
* oh, and we're targeting solr version 9.7.0, right?
this only does the last task you mentioned AFAICT though
2025-05-20 14038, 2025
julian45[m]
yeah afaict the only net-new commits on that branch besides that would update the submodules for the solr and zookeeper roles to a commit that's newer than they may currently be, but still older than where they need to be
2025-05-20 14057, 2025
julian45[m]
s/commits/commit/
2025-05-20 14038, 2025
LupinIII joined the channel
2025-05-20 14059, 2025
julian45[m]
i anticipate there will be little major movement here until yvanzo wakes up anyway, so i will brb shortly then proceed with deciphering the next task in the block yvanzo specified
alright, next task, the one titled "Upload updated configuration sets to ZooKeeper":... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/KCJEboUePFUCcdAoutUlkxoD>)
2025-05-20 14027, 2025
julian45[m]
* first, a note that this whole block that yvanzo has asked about will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-an… is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/PsyVyhacibCrlyfifZpAVoyc>)
2025-05-20 14039, 2025
julian45[m]
finally, the last top-level task within yvanzo's requested block, labeled "Create and reload Solr collections", is itself a block of three tasks that do some api calls. i'll break each of them down below.
2025-05-20 14038, 2025
julian45[m]
* api calls, all of which run as the `solr` user (again, default of `solr_user`) on the target server. i'll
2025-05-20 14020, 2025
mrnelgin is now known as nelgin
2025-05-20 14035, 2025
julian45[m]
* finally, the last top-level task within yvanzo's requested block, labeled "Create and reload Solr collections", is itself a block of three tasks that do some api calls, all of which run as the `solr` user (again, default of `solr_user`) on the target server. i'll break each of them down below.
the first subtask in the block, "List Solr collections", makes a GET request to a variably-defined url that _should_ resolve to `http://127.0.0.1:8983/api/cluster` (again, relative to the target server) based on [these defaults](https://github.com/metabrainz/ansible-r….... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/UtMSjdRAcCOJUeXhrOXACrBS>)
2025-05-20 14036, 2025
julian45[m]
* in that this task, even if it could successfully run, is not pulling in the latest work in mbsssss; it's pulling in an old commit
2025-05-20 14035, 2025
julian45[m]
* first, a note that this whole block that yvanzo has asked about will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-an… is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/McJdbYQfMzRhhPqzcVFTzDIw>)
2025-05-20 14005, 2025
julian45[m]
* so, absent my concern about the copy_links and links param values potentially stepping on each other, that task seems like it should run at this time, though it will likely not produce desired results due to the outdated submodule
2025-05-20 14029, 2025
julian45[m]
next, "Create Solr collections" loops through [`solr_collections`](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L107-L187) and, for each top-level collection definition within the dict that was NOT part of the collections returned by the server in the previous subtask... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/eyHmkVJzmCovbzCgpysaILiF>)
2025-05-20 14042, 2025
julian45[m]
* next, "Create Solr collections" loops through [`solr_collections`](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L107-L187) and, for each top-level collection definition within the dict that was NOT part of the collections returned by the server in the previous subtask,... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/dwaSvtGsFcVqhErAFRhIaOhb>)
2025-05-20 14035, 2025
julian45[m]
* next, "Create Solr collections" loops through [`solr_collections`](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L107-L187) and, for each top-level collection definition within the dict that was NOT part of the collections returned by the server in the previous subtask,... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/QUZcjSAOphlqvQyjhUnYACwj>)
2025-05-20 14048, 2025
julian45[m]
finally, the subtask named "Reload Solr collections" will loop through `_solr_configset_sync` (this is the output of the very first task in yvanzo's block, remember) and, for each collection that had a change as a result of the rsync task loop, will trigger a reload of that connection by calling out to `http://127.0.... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/gHhAGaUKwEbgRizDjQbAkOYh>)
2025-05-20 14010, 2025
julian45[m]
alright, that should cover the big block of ansible yvanzo asked for!... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/vSzAeaJFhlmnwTNPHicziwQc>)
2025-05-20 14017, 2025
d4rk-ph0enix joined the channel
2025-05-20 14025, 2025
d4rkie has quit
2025-05-20 14027, 2025
pavlovsfrog has quit
2025-05-20 14049, 2025
pavlovsfrog joined the channel
2025-05-20 14045, 2025
pavlovsfrog6 joined the channel
2025-05-20 14044, 2025
pavlovsfrog has quit
2025-05-20 14045, 2025
pavlovsfrog6 is now known as pavlovsfrog
2025-05-20 14045, 2025
pavlovsfrog8 joined the channel
2025-05-20 14008, 2025
pavlovsfrog has quit
2025-05-20 14008, 2025
pavlovsfrog8 is now known as pavlovsfrog
2025-05-20 14039, 2025
Kladky joined the channel
2025-05-20 14013, 2025
HemangMishra[m] has quit
2025-05-20 14002, 2025
rozlav8 has quit
2025-05-20 14017, 2025
rozlav8 joined the channel
2025-05-20 14047, 2025
yvanzo[m]
Back in search of a solution 🌄
2025-05-20 14035, 2025
yvanzo[m]
I had local changes to metabrainz-ansible, which I just pushed to the branch mb-solr-400.
2025-05-20 14033, 2025
yvanzo[m]
Looks like the same changes as in mb-solr-4.1.0 but the download URL, even though I already updated Solr version to 9.7.0 for each node yesterday.
2025-05-20 14043, 2025
yvanzo[m]
The task solr::install was successful, not the two others which are needed to update configsets and collections in SolrCloud.
2025-05-20 14000, 2025
Maxr1998 joined the channel
2025-05-20 14032, 2025
Maxr1998_ has quit
2025-05-20 14026, 2025
yvanzo[m]
I think I slashed it.
2025-05-20 14035, 2025
yvanzo[m]
(the issue)
2025-05-20 14041, 2025
lucifer[m]
yvanzo: hi! i am around if you need any help with the solr/search stuff.
2025-05-20 14050, 2025
yvanzo[m]
Hi lucifer: I successfully updated configsets and reloaded collections with MB Solr 4.1.0 in our SolrCloud 9 cluster, then updated the indexer to SIR 4.0.0.
2025-05-20 14012, 2025
lucifer[m]
awesome.
2025-05-20 14042, 2025
yvanzo[m]
There are 350K messages in Redis search.failed queue that need to be put back to search.index.
2025-05-20 14007, 2025
Maxr1998_ joined the channel
2025-05-20 14028, 2025
Maxr1998 has quit
2025-05-20 14054, 2025
yvanzo[m]
I have to fix the triggers that need to be prefixed with solr9_.
2025-05-20 14037, 2025
yvanzo[m]
Started to update search indexes, that will take some hours.
2025-05-20 14036, 2025
yvanzo[m]
Meanwhile redis is expected to pile search.index messages up (due to MB edits).
2025-05-20 14057, 2025
yvanzo[m]
Those will be processed only after the general update.
2025-05-20 14020, 2025
_BrainzGit
[listenbrainz-server] 14MonkeyDo merged pull request #3269 (03master…modal-open): Added a checkbox to make multiple listens or albums keeping the modal open https://github.com/metabrainz/listenbrainz-server…
2025-05-20 14037, 2025
yvanzo[m]
lucifer: Help would be welcome with avoiding SQL statement timeouts while SIR is reindexing everything.
2025-05-20 14045, 2025
lucifer[m]
[@yvanzo:chatbrainz.org](https://matrix.to/#/@yvanzo:chatbrainz.org) hi! sure what do I need to do?
Is it an issue with SIR config or with limitations set to our PG instance?
2025-05-20 14022, 2025
LupinIII
hi guyos! how's the schema change going? form what I lurked yesterday it went alright?
2025-05-20 14035, 2025
LupinIII
some new issues popped?
2025-05-20 14035, 2025
yvanzo[m]
(config at the bottom)
2025-05-20 14046, 2025
LupinIII
*popped up
2025-05-20 14001, 2025
LupinIII
question: is it inadvisable to work on instruments today or should that work fine?
2025-05-20 14042, 2025
yvanzo[m]
Search results won't be updated with recent edits today.
2025-05-20 14006, 2025
yvanzo[m]
The rest went fine.
2025-05-20 14014, 2025
yvanzo[m]
lucifer: Any clue about it?
2025-05-20 14046, 2025
yvanzo[m]
Importing artist, release, url… failed so far.
2025-05-20 14029, 2025
yvanzo[m]
And I guess recording would fail too, those are the only failing collections.
2025-05-20 14036, 2025
lucifer[m]
[@yvanzo:chatbrainz.org](https://matrix.to/#/@yvanzo:chatbrainz.org) oh weird. I'll take a look at it in 20 mins.
2025-05-20 14040, 2025
monkey[m]
lucifer: Can I put the bootstrap5 branch on beta.LB for a week or two? Would need to keep it there for user testing, but happy to keep it up to date with master during that time.
2025-05-20 14048, 2025
yvanzo[m]
lucifer: Also the live indexer is having a few errors such as `cannot serialize 19 (type int)`.
2025-05-20 14027, 2025
yvanzo[m]
(Those messages currently end up in the search.retry queue of /search-index-rebuilder-solr9 vhost.)
2025-05-20 14030, 2025
lucifer[m]
monkey: sure
2025-05-20 14034, 2025
lucifer[m]
yvanzo: on it
2025-05-20 14026, 2025
yvanzo[m]
I have to go in ~15min.
2025-05-20 14025, 2025
Kladky_ joined the channel
2025-05-20 14009, 2025
texke joined the channel
2025-05-20 14017, 2025
yvanzo[m]
lucifer: please have a look at `sir-solr9-prod` logs on `rakim`, will be back!
2025-05-20 14001, 2025
Kladky has quit
2025-05-20 14001, 2025
Kladky_ is now known as Kladky
2025-05-20 14007, 2025
lucifer[m]
yvanzo: it seems like an error from greenlet
2025-05-20 14044, 2025
lucifer[m]
i'll try to pull messages from search.retry queue and try to index them on a mb docker replica on trille to try reproduce and debug the error.
2025-05-20 14009, 2025
lucifer[m]
i see a new release for mb-docker hasn't been made yet?
2025-05-20 14039, 2025
lucifer[m]
yvanzo: my bad, i was looking the py2 version. the error is not from greenlet lxml.
Hi. Is there an issue with indexing in the Schema 30 release? I have upgraded and rebuilt the database from the full export and I'm now attempting to reindex. However, it's consistently stalling (not failing, just not doing anything) when it gets to the artist entity. Is this a known issue?
2025-05-20 14056, 2025
mayhem[m]
mglubb: Yes, I know that the team has been battling some indexing issues. How are things yvanzo ?
2025-05-20 14033, 2025
yvanzo[m]
Hi mglubb: Yes there are some issues. However, the mirrors will be able to either update to Solr 9 or keep Solr 7.
2025-05-20 14028, 2025
yvanzo[m]
lucifer: The issue isn’t with mb-docker, it is with our main instance.
2025-05-20 14052, 2025
bitmap[m]
both sir instances are connecting to the database in the some way, and we weren't setting any statement_timeout in the old sir, AFAICT, unless sqlalchemy was doing that automatically. so my best guess is that sir v4 is generating different (slower) queries
2025-05-20 14001, 2025
bitmap[m]
s/some/same/, s//`/, s//`/
2025-05-20 14046, 2025
mglubb[m]
Thanks, yvanzo , mayhem . I see you have an issue with ongoing indexing of changes. My issue is that full indexing is extremely slow. I've already been using Solr 9 for some time and have rebuilt indices without issue. The difference appears to be the schema change / sir python 3 change perhaps?
2025-05-20 14048, 2025
yvanzo[m]
mglubb: Solr 9 backups are available already, to spare you the cost of full indexing.
2025-05-20 14003, 2025
mglubb[m]
I saw those thanks, yvanzo . However, the latest is from the 17th May.
2025-05-20 14017, 2025
yvanzo[m]
I did test full indexing over the week-end and didn't notice any issue though, but that wasn't the same host.
2025-05-20 14058, 2025
yvanzo[m]
Right, there is no backup available since the schema change indeed.
2025-05-20 14049, 2025
yvanzo[m]
It should be available tomorrow, as we are hoping to update all indexes today.
2025-05-20 14001, 2025
yvanzo[m]
bitmap: It was working in mb-docker though, so I suspect a setup issue.
2025-05-20 14008, 2025
mglubb[m]
Thanks, yvanzo . I'll use the backups for now but it would give me a nice warm fuzzy feeling to know that I can re-index our mirror in an emergency.
2025-05-20 14002, 2025
yvanzo[m]
It took ~12h during tests.
2025-05-20 14032, 2025
bitmap[m]
yvanzo[m]: do we configure `statement_timeout` for PG in mb-docker, though? because it defaults to 0. in production it's set to 55s, IIRC
2025-05-20 14028, 2025
yvanzo[m]
bitmap: Is there an easy way to change this timeout for sir until tomorrow?
2025-05-20 14013, 2025
yvanzo[m]
(SIR is setting a timeout for RabbitMQ, not for PG afaict.)
2025-05-20 14049, 2025
bitmap[m]
we could change sir to set statement_timeout=0 when it opens a session, or we could add a sir role to PG and configure a statement_timeout for it
2025-05-20 14058, 2025
yvanzo[m]
There is no statement_timeout set in MB Docker either.
2025-05-20 14008, 2025
yvanzo[m]
Adding a role to en existing PG user should be fine?
2025-05-20 14028, 2025
bitmap[m]
ok, I can add sir_ro role based on musicbrainz_ro
2025-05-20 14029, 2025
bitmap[m]
actually we already have a sir role -- not sure what this was used for
2025-05-20 14023, 2025
bitmap[m]
yvanzo: you should be able to connect as the `sir` user then. I just set `statement_timeout = 0` on the role
2025-05-20 14037, 2025
bitmap[m]
it was already granted musicbrainz_ro previously
2025-05-20 14031, 2025
yvanzo[m]
Thank you, on updating indexes again.
2025-05-20 14039, 2025
mayhem[m]
18 | 10cc | 937655 | Ten CC | 1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/iitiUNaSOgyQtHQMkQOkXxCK>)
2025-05-20 14040, 2025
lucifer[m]
yvanzo: yes i wanted to use it for running a patch.
2025-05-20 14048, 2025
mayhem[m] cries on his sql prompt
2025-05-20 14012, 2025
lucifer[m]
yvanzo, bitmap: i think i have found some inefficiences in the artist core which could speedup the indexing a few times. that might make state_timeout unneeded.
2025-05-20 14035, 2025
lucifer[m]
* yvanzo, bitmap: i think i have found some inefficiences in the artist core which could speedup the indexing a few times. that might make disabling statement\_timeout unneeded.
2025-05-20 14017, 2025
bitmap[m]
what did you find?
2025-05-20 14001, 2025
yvanzo[m]
a mole
2025-05-20 14048, 2025
lucifer[m]
column_property properties are loaded eagerly for entities which don't need them. for instance, place_count, label_count, artist_count is being calculated for each artists' begin area/end area/area etc.
