#metabrainz

first, a note that this whole block will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L23) is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/hoFHDgPhaqLiVUdjSzFWClyL>)

hmm, i think i see part of the problem here

in that this task, even if it could successfully run, is not pulling in the latest work in mbssss; it's pulling in an old commit

s/mbssss/mbsssss/

if we look [one level above the local directory we want to pull config directories from], we'll see that its copy of the mbssss dir is pulling in [mbsssss @ e2e630a](https://github.com/metabrainz/mbsssss/tree/e2e630a8eac72457ad085d735043e787c9b076e0), which is nearly a year old... the submodule has not been updated in metabrainz-ansible for some time

* if we look [one level above the "local" (to the ansible control node) directory that task wants to pull config directories from](https://github.com/metabrainz/metabrainz-ansible/tree/main/files/solr_cluster_servers), we'll see that its copy of the mbssss dir is pulling in [mbsssss @ e2e630a](https://github.com/metabrainz/mbsssss/tree/e2e630a8eac72457ad085d735043e787c9b076e0), which is nearly a year old... the submodule has not been

updated in metabrainz-ansible for some time

indeed, nice find, I'll open a PR to update that

so, absent my concern about the copy_links and links params potentially stepping on each other, that task should run at this time, though it will likely not produce desired results due to the outdated submodule

* so, absent my concern about the copy_links and links params potentially stepping on each other, that task seems like it should run at this time, though it will likely not produce desired results due to the outdated submodule

* so, absent my concern about the copy_links and links param values potentially stepping on each other, that task seems like it should run at this time, though it will likely not produce desired results due to the outdated submodule

bitmap[m]: fwiw, yvanzo had opened a PR to update all of the submodules in metabrainz-ansible ~1 month ago, but that still wouldn't have caught the work done in the past few weeks

* first, a note that this whole block will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L23) is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/BUYPoWljJYzhSmetWtXcYdLZ>)

yeah, looks like we need to update the .jar file (solr_shared_libraries) too

oh yes, and have solr_shared_libraries point to metabrainz/mb-solr rather than yvanzo's fork

I guess the intent of his previous PR was to only update the roles/ anyway (but conveniently there were no mbsssss changes at the time, that's the only non-role submodule)

yeah, looks like it

oh, and solr_version should be 9.7.0, right?

* oh, and we're targeting solr version 9.7.0, right?

if so, we need to update `solr_version` in [here](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml), as well as the checksum for the download

this only does the last task you mentioned AFAICT though

yeah afaict the only net-new commits on that branch besides that would update the submodules for the solr and zookeeper roles to a commit that's newer than they may currently be, but still older than where they need to be

s/commits/commit/

i anticipate there will be little major movement here until yvanzo wakes up anyway, so i will brb shortly then proceed with deciphering the next task in the block yvanzo specified

thanks for your help so far, I'll be AFK for a bit too, but I've opened https://github.com/metabrainz/metabrainz-ansibl... in the meantime :)

alright, next task, the one titled "Upload updated configuration sets to ZooKeeper":... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

* first, a note that this whole block that yvanzo has asked about will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L23) is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/PsyVyhacibCrlyfifZpAVoyc>)

finally, the last top-level task within yvanzo's requested block, labeled "Create and reload Solr collections", is itself a block of three tasks that do some api calls. i'll break each of them down below.

* api calls, all of which run as the `solr` user (again, default of `solr_user`) on the target server. i'll

* finally, the last top-level task within yvanzo's requested block, labeled "Create and reload Solr collections", is itself a block of three tasks that do some api calls, all of which run as the `solr` user (again, default of `solr_user`) on the target server. i'll break each of them down below.

documentation for the `ansible.builtin.uri` module used in this block can be found [here](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/uri_module.html).

the first subtask in the block, "List Solr collections", makes a GET request to a variably-defined url that _should_ resolve to `http://127.0.0.1:8983/api/cluster` (again, relative to the target server) based on [these defaults](https://github.com/metabrainz/ansible-role-solr/blob/main/defaults/main.yml#L123-129).... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/UtMSjdRAcCOJUeXhrOXACrBS>)

* in that this task, even if it could successfully run, is not pulling in the latest work in mbsssss; it's pulling in an old commit

* first, a note that this whole block that yvanzo has asked about will only run on the server defined in `solr_cloud_leader`, which in [our case](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L23) is mb-solrcloud-1... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/McJdbYQfMzRhhPqzcVFTzDIw>)

* so, absent my concern about the copy_links and links param values potentially stepping on each other, that task seems like it should run at this time, though it will likely not produce desired results due to the outdated submodule

next, "Create Solr collections" loops through [`solr_collections`](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L107-L187) and, for each top-level collection definition within the dict that was NOT part of the collections returned by the server in the previous subtask... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

* next, "Create Solr collections" loops through [`solr_collections`](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L107-L187) and, for each top-level collection definition within the dict that was NOT part of the collections returned by the server in the previous subtask,... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

* next, "Create Solr collections" loops through [`solr_collections`](https://github.com/metabrainz/metabrainz-ansible/blob/main/group_vars/solr_servers/solr.yml#L107-L187) and, for each top-level collection definition within the dict that was NOT part of the collections returned by the server in the previous subtask,... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

finally, the subtask named "Reload Solr collections" will loop through `_solr_configset_sync` (this is the output of the very first task in yvanzo's block, remember) and, for each collection that had a change as a result of the rsync task loop, will trigger a reload of that connection by calling out to `http://127.0.... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

alright, that should cover the big block of ansible yvanzo asked for!... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

Back in search of a solution 🌄

I had local changes to metabrainz-ansible, which I just pushed to the branch mb-solr-400.

Looks like the same changes as in mb-solr-4.1.0 but the download URL, even though I already updated Solr version to 9.7.0 for each node yesterday.

The task solr::install was successful, not the two others which are needed to update configsets and collections in SolrCloud.

I think I slashed it.

(the issue)

yvanzo: hi! i am around if you need any help with the solr/search stuff.

Hi lucifer: I successfully updated configsets and reloaded collections with MB Solr 4.1.0 in our SolrCloud 9 cluster, then updated the indexer to SIR 4.0.0.

awesome.

There are 350K messages in Redis search.failed queue that need to be put back to search.index.

I have to fix the triggers that need to be prefixed with solr9_.

Started to update search indexes, that will take some hours.

Meanwhile redis is expected to pile search.index messages up (due to MB edits).

Those will be processed only after the general update.

[listenbrainz-server] 14MonkeyDo merged pull request #3269 (03master…modal-open): Added a checkbox to make multiple listens or albums keeping the modal open https://github.com/metabrainz/listenbrainz-serv...

lucifer: Help would be welcome with avoiding SQL statement timeouts while SIR is reindexing everything.

[@yvanzo:chatbrainz.org](https://matrix.to/#/@yvanzo:chatbrainz.org) hi! sure what do I need to do?

See https://gist.github.com/yvanzo/1bfe92a27fa07ea0...

Is it an issue with SIR config or with limitations set to our PG instance?

hi guyos! how's the schema change going? form what I lurked yesterday it went alright?

some new issues popped?

(config at the bottom)

*popped up

question: is it inadvisable to work on instruments today or should that work fine?

Search results won't be updated with recent edits today.

The rest went fine.

lucifer: Any clue about it?

Importing artist, release, url… failed so far.

And I guess recording would fail too, those are the only failing collections.

[@yvanzo:chatbrainz.org](https://matrix.to/#/@yvanzo:chatbrainz.org) oh weird. I'll take a look at it in 20 mins.

lucifer: Can I put the bootstrap5 branch on beta.LB for a week or two? Would need to keep it there for user testing, but happy to keep it up to date with master during that time.

lucifer: Also the live indexer is having a few errors such as `cannot serialize 19 (type int)`.

(Those messages currently end up in the search.retry queue of /search-index-rebuilder-solr9 vhost.)

monkey: sure

yvanzo: on it

I have to go in ~15min.

lucifer: please have a look at `sir-solr9-prod` logs on `rakim`, will be back!

yvanzo: it seems like an error from greenlet

i'll try to pull messages from search.retry queue and try to index them on a mb docker replica on trille to try reproduce and debug the error.

i see a new release for mb-docker hasn't been made yet?

yvanzo: my bad, i was looking the py2 version. the error is not from greenlet lxml.

* from greenlet but lxml.

[sir] 14amCap1712 opened pull request #164 (03master…fix-id-delete): doc_id for pysolr.delete should be a string https://github.com/metabrainz/sir/pull/164