I stopped 5, I'll restart it once 4 recovered (if it does)
2024-05-27 14858, 2024
rimskii[m] joined the channel
2024-05-27 14858, 2024
rimskii[m]
<lucifer> "rimskii: hi! let me know when..." <- lucifer (IRC): hi! srry, i saw your message just now, was on flight.
2024-05-27 14858, 2024
rimskii[m]
is it okay if I will ask the questions here? :>
2024-05-27 14810, 2024
mayhem
yes
2024-05-27 14830, 2024
mayhem
rimskii[m]: just ask questions; we may not answer immediately, but we do eventually. :)
2024-05-27 14826, 2024
zas
I stopped sir-prod for now
2024-05-27 14850, 2024
zas
restarting solr 5
2024-05-27 14828, 2024
zas
lucifer: the main question is why solr 5 eventually start to read from disk very heavily (like 800MB/s vs 50MB/s for others), when it does, the cluster crashes at some point, I think that's because solr5 gets very slow (load > 100) and it somehow cascades
lucifer (IRC): I'm setting up my stuff now. wanted to ask what about the apple music and soundcloud apis? I have an account to use soundcloud APIs, but not for the musickit one( also its paid. do we have a MB acc for it or should I buy it?
2024-05-27 14830, 2024
lucifer
rimskii[m]: we have an apple music developer token you should be able to use methinks.
2024-05-27 14831, 2024
rimskii[m]
cool !
2024-05-27 14857, 2024
zas
can it be an underlying hardware issue? I mean that's really weird it always happen on solr5 (and sometimes 4) but never on others
2024-05-27 14837, 2024
rimskii[m]
I think i will start from importing functions first (for Spotify, Soundcloud and apple music), then go to exports
2024-05-27 14854, 2024
zas
I could snapshot the vm, destroy it and recreate it while preserving IPs (I think)
2024-05-27 14831, 2024
zas
but first let's see if lucifer can get something from logs
2024-05-27 14821, 2024
zas
bumping the rate limit to 60 (from 20/s)
2024-05-27 14849, 2024
mayhem
monkey[m]: lucifer : do we know why the feed doesn't load?
lucifer (IRC):... (full message at <https://matrix.moviebrainz.org/_matrix/media/v3/download/moviebrainz.org/KYKBHHtNxHeGnJNJWCqYqfHN>)
2024-05-27 14812, 2024
lucifer
zas: looks like memory issues. every 1s, GC activity is causing 0.3s stalls
2024-05-27 14815, 2024
zerodogg has quit
2024-05-27 14839, 2024
mayhem
aerozol had the same problem.
2024-05-27 14846, 2024
lucifer
rimskii[m]: i see, makes sense. i can help with it in a while.
2024-05-27 14839, 2024
rimskii[m]
okay, thanks!
2024-05-27 14829, 2024
lucifer
when that happens solr starts reading a lot of data from disks.
2024-05-27 14834, 2024
rimskii[m]
also if I want to PR should I first open a ticket for it? if the PR is related to GSoC. I would like to commits PR little by little
2024-05-27 14821, 2024
mayhem
rimskii[m]: we open tickets sometimes, but in a lot of cases they are not needed for ordinary PRs. I suspect that lucifer wants you to open a ticket, he'll ask you to do that.
2024-05-27 14845, 2024
mayhem
.. *if* lucifer wants ....
2024-05-27 14827, 2024
zas
lucifer: by memory issues do you mean lack of RAM? or something else?
2024-05-27 14846, 2024
lucifer
zas: yes. RAM and JVM heap.
2024-05-27 14801, 2024
lucifer
but still not sure why only on two particular nodes.
2024-05-27 14835, 2024
zas
perhaps they are slower than others (shared cpu) and it leads to the issue more quickly
2024-05-27 14854, 2024
zas
or an underlying hardware issue leads to that
2024-05-27 14854, 2024
lucifer
possible yeah
2024-05-27 14811, 2024
zas
I think I'll rescale this server: it wil rule out an underlying hardware issue and give it more ram. Extra cost until we moved to new cluster though.
2024-05-27 14826, 2024
mayhem
fine.whatever.
2024-05-27 14832, 2024
zas
let's try
2024-05-27 14845, 2024
zas
I'll stop solr5
2024-05-27 14849, 2024
pranav[m] joined the channel
2024-05-27 14849, 2024
pranav[m]
akshaaatt (IRC): if ur free post today’s dev meet id like to have a small brief meet re how to proceed with my GSoC project if that’s fine with u..
2024-05-27 14832, 2024
akshaaatt
Sure, pranav[m] !
2024-05-27 14837, 2024
lucifer
mayhem: i'll try to debug. sentry has been down for a few days so can't check ther.
2024-05-27 14854, 2024
zas
solr5 has twice more cores now, but I couldn't get more RAM. Though underlying hardware changed at least. I'll slowly set the rate limit back to normal
2024-05-27 14810, 2024
mayhem
ok, no rush I would say. not sure how many people use that page.
2024-05-27 14817, 2024
lucifer
can you try loading it now?
2024-05-27 14829, 2024
lucifer
(no fixes just to see if something is logged in prod)
2024-05-27 14846, 2024
pranav[m]
lucifer (IRC): for a project of mine that I have to make for my college, I had a doubt re pagination from backend. I have let’s say 10,000-15,000 entries and am using page number pagination.. what shud be my page size and how is it determined in order for the server to not slow down
pranav[m]: no fixed rule for that, depends on what database you are using, how your data is structured, how often data is retrieved and lots of other factors.
2024-05-27 14812, 2024
lucifer
probably start so that most common cases are covered in 1 api call. if that is too much then decrease the page count.
2024-05-27 14830, 2024
lucifer
but to be honest, most applications never reach that scale that you need to worry about such things.
2024-05-27 14857, 2024
pranav[m]
Okay, thanks a lot lucifer_ (IRC) will keep this in mind
2024-05-27 14821, 2024
lucifer
rimskii[m]: PRs are fine. tickets are not really required in most cases but if you like to create some structure or etc. feel free to.
2024-05-27 14824, 2024
lucifer for the troi part. https://github.com/metabrainz/troi-recommendation-playground/blob/main/troi/tools/spotify_lookup.py
2024-05-27 14857, 2024
lucifer
this is an example of how we export playlists from LB to spotify. you can create a similar function here that imports playlists from spotify to LB.
2024-05-27 14815, 2024
lucifer
most of the code you have in LB server should remain the same just live in this repo instead.
2024-05-27 14812, 2024
rimskii[m]
got it, thank you very much !
2024-05-27 14802, 2024
zerodogg joined the channel
2024-05-27 14845, 2024
rimskii[m]
for the last:... (full message at <https://matrix.moviebrainz.org/_matrix/media/v3/download/moviebrainz.org/fcVFRrbLcFxoWGulOakCnEsZ>)
2024-05-27 14830, 2024
twodoorcoupe joined the channel
2024-05-27 14803, 2024
zas
Hey twodoorcoupe
2024-05-27 14820, 2024
twodoorcoupe
zas: what's up
2024-05-27 14803, 2024
zas
a lot on Picard front ;) check latest merged PRs
2024-05-27 14818, 2024
zas
and trying to restore our search cluster atm
2024-05-27 14859, 2024
zas
btw, solr cluster is back, new solr5 has twice cores, so it handles twice more requests than others, everything is back to normal
2024-05-27 14813, 2024
lucifer
rimskii[m]: you can finish moving the spotify import for now. i'll look into soundcloud apis and let you know how we should proceed
2024-05-27 14847, 2024
zas
twodoorcoupe: if you have any question about Picard, feel free to ask.
2024-05-27 14804, 2024
twodoorcoupe
zas: thank you, I saw you have been doing a ton of refactoring lately
2024-05-27 14850, 2024
rimskii[m]
<lucifer> "rimskii: you can finish moving..." <- okay, thanks a lot! ll let you know once I finish or mybe need a help xd
2024-05-27 14800, 2024
lucifer
sure sounds good
2024-05-27 14803, 2024
zas
Yes, we are heading towards Picard3 and there were a lot of things we couldn't do before without breaking too much stuff
2024-05-27 14833, 2024
zas
Picard3 will come with new plugin system and so compatibility will be broken anyway. outsidecontext is also experimenting moving to PySide6.
2024-05-27 14835, 2024
twodoorcoupe
zas: I was planning to use extension points for registering processing functions. If that is bound to change I can modify it later on
2024-05-27 14832, 2024
zerodogg has quit
2024-05-27 14854, 2024
zas
The main change is we'll make plugin accessible extension points much more obvious, and we'll provide a "plugin API" to guarantee access to Picard internals.
2024-05-27 14837, 2024
relaxoMob joined the channel
2024-05-27 14835, 2024
zas
In the past, maintaining plugins compat was always a bit of a problem, because plugins were accessing the Picard code freely. It will be still possible (that's Python), but if we break something not part of the "official" plugin API it'll be not our problem ;)
2024-05-27 14831, 2024
zas
Basically every patch had an underlying question: does it break one plugin? -> look for plugins code, etc... kinda impossible to manage with a growing list of plugins
2024-05-27 14841, 2024
zas
For your GSoC project, I think you can work on a PR over Picard code, without bothering plugins side while keeping in mind it will be a plugin at the end.
2024-05-27 14853, 2024
zas
The new plugin system should be out (I hope) before end of GSoC. I'll focus on it once the cleanup work will be done.
2024-05-27 14803, 2024
twodoorcoupe
Makes sense, thank you. Do you and outsidecontext prefer to see a pr for each feature added, let's say each week, or a larger one every once in a while?
2024-05-27 14845, 2024
twodoorcoupe
Great, I had already planned to work on plugins side in the final weeks
2024-05-27 14853, 2024
zas
We usually prefer to follow devel steps, so opening a draft PR early is often better: we can review and give you feedback along the way
2024-05-27 14806, 2024
outsidecontext
hi
2024-05-27 14816, 2024
zas
Hey outsidecontext
2024-05-27 14819, 2024
twodoorcoupe
Ok, will do
2024-05-27 14836, 2024
outsidecontext
twodoorcoupe: ideal would be multiple PRs, but for parts that make sense individually. doesn't need to be exactly weekly or such.
2024-05-27 14806, 2024
zas
yes, subdivide your work in small reviewable bits
2024-05-27 14813, 2024
outsidecontext
and you can open PRs already early as soon as you think there is something to discuss. Mark the PR as a draft and we can have an active discussion on anything
2024-05-27 14838, 2024
twodoorcoupe
Perfect
2024-05-27 14845, 2024
outsidecontext
we currently had multiple of those due to zas' many refactorings :)
2024-05-27 14859, 2024
mayhem
<BANG>
2024-05-27 14822, 2024
mayhem
hello everyone. I'm a poor excuse for the usual spaniard running the show!
Last week mostly played around with feed generation in Python with my feed reader. Found that Payoneer blocked me for not using Gmail during registration and luckily I fixed it :)
2024-05-27 14830, 2024
mayhem
And finally JadeBlueEyes:
2024-05-27 14842, 2024
mayhem
Nothing much from me from last week! I've been very busy with exams, but I've only got three more to go this week!
2024-05-27 14842, 2024
mayhem
After that, I'll work on setting up a local mock for testing sending emails
2024-05-27 14803, 2024
mayhem
That was it for mailed in reviews, lets jump to reviews for the rest of us.
Also worked on SIR testing and Solr 9.6 update, upgraded the wiki server to Noble, plus some support and tickets triage.
2024-05-27 14855, 2024
yvanzo
Good luck to the IA!
2024-05-27 14803, 2024
yvanzo
Fin, go atj?
2024-05-27 14807, 2024
atj
hello
2024-05-27 14840, 2024
atj
Last week I continued testing and tweaking the new Solr cluster. I configured ZooKeeper ACLs, wrote some scripts to try and balance leaders across the nodes in the cluster, improved the Ansible role and started working on documentation.
2024-05-27 14836, 2024
atj
I also worked with yvanzo to prepare the cluster for use as the search provider on beta.
2024-05-27 14805, 2024
atj
and bootstrapped the new Matrix server, and assisted lucifer in setting up Borg backups.
2024-05-27 14836, 2024
atj
that's all I can remember at this point! lucifer?
2024-05-27 14844, 2024
lucifer
sure sounds good!
2024-05-27 14847, 2024
lucifer
hi all!
2024-05-27 14848, 2024
lucifer
last week, i worked on setting up chatbrainz, participated in the gsoc intro meeting, worked on existing LB PRs to fix dump issues, and speedup imports in spark.
2024-05-27 14814, 2024
lucifer
also worked on exploring setting up spark cluster with ansible and some misc LB bugs.
2024-05-27 14823, 2024
lucifer
that's it for me. mayhem next?
2024-05-27 14829, 2024
mayhem
hey o
2024-05-27 14841, 2024
mayhem
Annoying week, dealing with lots of stupid stuff. like Hetzer abuse BS -- more on that during that topic meeting.
2024-05-27 14856, 2024
mayhem
I worked a bit on JSPF cleanup, since we identified a few ways in which we didn't meet the spec.
2024-05-27 14826, 2024
mayhem
Had two conference calls, one with researchers who are investigating the fairness of recommendation engines and they are
2024-05-27 14826, 2024
mayhem
interesting in working on that with us -- which is quite interesting. The other call was the GSoC welcome meeting which is all around lovely.
2024-05-27 14848, 2024
mayhem
This week I've got lots of small stuff to look after and then I can finally dig back into LB Local.
2024-05-27 14854, 2024
mayhem
fin. twodoorcoupe go!
2024-05-27 14802, 2024
twodoorcoupe
hello folks!
2024-05-27 14832, 2024
twodoorcoupe
last week I finished preparing drafts for my gsoc project
2024-05-27 14800, 2024
twodoorcoupe
started working on filtering out cover art images by size
