[bookbrainz-site] 14MonkeyDo merged pull request #1106 (03import-entities…import-annotation-id): chore: Allow annotation revision ids to be null (rebased) https://github.com/metabrainz/bookbrainz-site/pul…
2024-07-17 19917, 2024
Kladky has quit
2024-07-17 19931, 2024
Kladky joined the channel
2024-07-17 19914, 2024
rimskii[m] has quit
2024-07-17 19959, 2024
pranav[m] has quit
2024-07-17 19959, 2024
discordbrainz has quit
2024-07-17 19913, 2024
discordbrainz joined the channel
2024-07-17 19956, 2024
twodoorcoupe[m] has quit
2024-07-17 19927, 2024
atj[m] has quit
2024-07-17 19905, 2024
outsidecontext[m has quit
2024-07-17 19914, 2024
outsidecontext[m joined the channel
2024-07-17 19953, 2024
BobSwift[m] has quit
2024-07-17 19908, 2024
kellnerd[m]
<mayhem[m]> "https://blog.metabrainz.org/2024..." <- It took me a while, but I just realized that the 23rd birthday post is from exactly a year ago and today we are already celebrating the 25th... time flies by 😁
2024-07-17 19935, 2024
mayhem[m]
wut?
2024-07-17 19947, 2024
mayhem[m]
math, my nemesis.
2024-07-17 19940, 2024
mayhem[m]
well, no one noticed for a whole year, so 🤷
2024-07-17 19956, 2024
theflash[m] has quit
2024-07-17 19940, 2024
atj[m] joined the channel
2024-07-17 19941, 2024
atj[m]
[@lucifer:chatbrainz.org](https://matrix.to/#/@lucifer:chatbrainz.org): it would be good to have the LB Solr cluster managed using Ansible, considering all the work done for the MB cluster
2024-07-17 19941, 2024
atj[m]
Is it a standard SolrCloud installation?
2024-07-17 19921, 2024
lucifer[m]
atj: there is not cluster yet, we are still experimenting to see if it performs better than typesense (what we have now). if it does then we will create a proper cluster for production. but yes makes sense
2024-07-17 19921, 2024
atj[m]
OK, well let me know how it goes. MB cluster seems to work well on ARM VMs which are cheaper and offers better performance.
2024-07-17 19936, 2024
Sophist-UK has quit
2024-07-17 19951, 2024
atj[m]
Don't set the Solr heap size too high would be my advice. Ideally you want to fit the indexes in page cache but I don't know how big they will be for LB
2024-07-17 19953, 2024
lucifer[m]
makes sense, will keep it in mind.
2024-07-17 19918, 2024
atj[m]
Are you storing documents in Solr or just IDs?
2024-07-17 19904, 2024
atj[m]
Storing the MB document XML in Solr was a mistake IMV, it's resulted in the indexes being much bigger than they need to be and reduces performance significantly
2024-07-17 19933, 2024
atj[m]
But then I don't know the whole history behind it.
2024-07-17 19937, 2024
mayhem[m] uploaded an image: (546KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/ugbPRzkrNHFCnDzqTPgdgqPa/image.png >
2024-07-17 19950, 2024
mayhem[m]
LOLOLOLOL, I feel seen.
2024-07-17 19955, 2024
leftmostcatUTC-7 joined the channel
2024-07-17 19955, 2024
leftmostcatUTC-7
I feel trapped.
2024-07-17 19904, 2024
lucifer[m]
<atj[m]> "Are you storing documents in..." <- JSON docs, only non searchable fields are mbids.
2024-07-17 19924, 2024
lucifer[m]
<atj[m]> "But then I don't know the..." <- indeed, i have discussed with yvanzo in the past to get rid of it.
2024-07-17 19925, 2024
lucifer[m]
i would like to get rid of it too for performance reasons, its slows down writing the response too imo.
2024-07-17 19942, 2024
lucifer[m]
so a perf win in many ways if we can get rid of it.
2024-07-17 19907, 2024
lucifer[m]
should probably restart that discussion now that solr 9 upgrade is done.
2024-07-17 19911, 2024
mayhem[m]
<atj[m]> "But then I don't know the..." <- Back in the day we favored not having to do another DB query to fetch the data. we accepted the larger indexes as a tradeoff for less load on our DB server. but we're in a much different place now. but scaling wise, hosting capabilities and money.
2024-07-17 19915, 2024
lucifer[m]
would actually simplify a lot of stuff in Sir too.
2024-07-17 19953, 2024
lucifer[m]
mayhem[m]: also it is XML centric.
2024-07-17 19906, 2024
lucifer[m]
XML requests are served directly iirc.
2024-07-17 19927, 2024
lucifer[m]
where as JSON requests need to deserialize the xml and reserialize it to JSON to serve the request
2024-07-17 19936, 2024
atj[m]
and it's a shitty Java dependency that nobody in their right mind should want to deal with
2024-07-17 19917, 2024
lucifer[m]
yeah fair and solr has built in xml support too iirc so if we want xml we can get that directly anyway.
