zas: you're traveling today, no? at least Solr appears to have been stable over the weekend.
iliekcomputers
moin!
ruaok
moin. reading #579 now. what should be next?
iliekcomputers
how do I connect to rabbitmq from the cluster?
one sec, let me see
i mean, what's the host/port
ruaok
we opened the rabbitmq port from leader to lemmy, I believe.
try lemmy.meb.o <rabbitmq port>
and if it works, document that in syswiki. :)
iliekcomputers
cool!
zas
ruaok: I'm back at home, and yes, SOLR looks stable, but i think the cause of all issues is network instability, communication between all nodes is critical, and there was a switch between lb3 & lb4 on saturday that was very likely caused by a network issue according to my research. I think it's out of our control (because that's underlying hetzner cloud network).
ruaok
ok
iliekcomputers
after 579, there are a few PRs unrelated to each other we can do based on priority
ruaok gets deja vu reviewing the incremental dumps PR
ruaok
I've reviewed this before.
iliekcomputers
hmm?
ruaok
in AB. :)
I see that in dumping full dumps and incremental dumps happens in two separate steps.
that is going to lead to inconsistent dumps.
iliekcomputers
i added a command to create full dumps consistent with the last incremental dump.
ruaok
if the incremental dumps are "keeping time" then dumping a incremental dump and a partial dump should happen at the same time. this ensures that the data in a full dump ends on the same boundary as the incremental dumps.
does `/code/listenbrainz/admin/create-dumps.sh full` invoke that mode?
I wonder if we should dump incrementals more frequently than once a week.
because this frequency defines how often we can update stats/recommendations.
iliekcomputers
every day?
ruaok
sundays and thursdays to match MB?
how intensive is it to dump an incremental?
iliekcomputers
I am not sure. it'll have to run on production for us to see.
akhilesh
Mr_Monkey: Hi!
ruaok
lets leave it for now and observe.
Mr_Monkey
Hi akhilesh
akhilesh
Mr_Monkey: I did some mistake on resolving conflict with master by using git rebase, please look in the PR
I confused with the solution
Mr_Monkey
What mistakes akhilesh ? I see the merge commits, but it's hard for me to spot an issue in there
Talking about package-lock, correct?
akhilesh
It shows merge commits also in my PR, how to remove them?
ruaok
iliekcomputers: can we run the incremental dumps on beta without interfering with the main dumps running on main?
we'd need to modify cron on the beta setup to avoid dumping full dumps...
Mr_Monkey
akhilesh: Right. Well, I suppose there's no harm done if you merged what was on master, it will just make the history a bit messier, but we can live with that.
iliekcomputers
yea, shouldn't be much work.
ruaok
I'd like to see that.
because I am not spotting any problems in this. makes me suspicious. lol.
D4RK-PH0ENiX has quit
iliekcomputers
we'll need to create a base full dump first, because not all listens have `inserted_timestamp` (initially we didn't save it).
or just use one of the existing dumps as the base dump, we keep track of dumps already.
Mr_Monkey
akhilesh: For future rebases, though, it would be good to figure out what happened. In theory, locally you should git fetch, then git rebase master, and once done, git push --force-with-lease (not a pull or regular push). I suppose you didn't force push?
ruaok
or we can turn off dumping on the main site and just run it in beta for a while until it is stable.
Mr_Monkey
In any case, I would say leave the PR as is
ruaok
... that actually makes the most sense, I think. and is easiest.
iliekcomputers
yeah, sounds good to me.
do we start the series at the beginning and truncate the existing dump table, or would the current table be okay?
ruaok
ideally we wouldn't break the sequence at all and just start emitting partial dumps as well.
probably not important since we don't have a lot of consumers, but it is a good practice.
iliekcomputers
cool, that should work.
we hadn't been emitting series numbers with previous dumps, i changed up filenames to include that.
akhilesh
Mr_Monkey: Yes I did not use force push, now how to correct this, it looks pretty bad in the PR?
ruaok
oh, interesting. I like how you did the file naming.
ok, push to beta, iliekcomputers. looks really nice!
Mr_Monkey
akhilesh: I would advise against trying to fix it. In my experience, it's a great way to make is worse :P
The commit history isn't too bad, and won't create merge conflicts, so I think the best way forward is to continue as is.
We've all borked our commit history before.
akhilesh
ok
Mr_Monkey
You could use `git reflog` locally to try and revert to before the rebase, then try to rebase again, but as I said I'm not sure it's worth the risk.
ruaok, yvanzo: i plan to upgrade SOLR from 7.5 to 7.7. According to change logs there should be no breaking changes for us. But a second sight is welcome.