Or simply just remove all the relationships from all the relaionship_sets where the deleted work is the target?
2020-01-28 02857, 2020
ruaok
iliekcomputers: my schedule is getting a bit mixed up for tonight. I'm free now, but 19h is more complicated. can we talk now?
2020-01-28 02834, 2020
iliekcomputers
Sure, I'm walking home, we can do it now
2020-01-28 02851, 2020
iliekcomputers
So
2020-01-28 02854, 2020
ruaok
shit, I didn't get a chance yet to read your PR. :(
2020-01-28 02801, 2020
iliekcomputers
Np
2020-01-28 02827, 2020
iliekcomputers
So there was a problem with us sending huge rmq messages
2020-01-28 02846, 2020
iliekcomputers
Which I fixed and deployed on the cluster
2020-01-28 02854, 2020
sarthak_jain joined the channel
2020-01-28 02803, 2020
sarthak_jain
Hi pristine__
2020-01-28 02804, 2020
iliekcomputers
I also added sentry and more exception handling.
2020-01-28 02835, 2020
iliekcomputers
At this point, I'm happy enough with the code to calculate and ship around stats
2020-01-28 02845, 2020
ruaok
excellent.
2020-01-28 02858, 2020
iliekcomputers
However the cluster doesn't get updated with new data regularly
2020-01-28 02859, 2020
ruaok
that means we're a step closer to a release, yes?
2020-01-28 02808, 2020
iliekcomputers
Yes
2020-01-28 02821, 2020
sarthak_jain
Since, I did not have an actual dataframe to check with, I created my own dataframe using the schema provided, and used the pyspark sql module queries to check if I am getting the desired results, which I am able to get.
2020-01-28 02829, 2020
ruaok
fortunately pristine__ has a PR that loads new data.
2020-01-28 02835, 2020
ruaok
not incrementals, but still.
2020-01-28 02844, 2020
sarthak_jain
But, when I run `spark_test.sh', I get errors.
2020-01-28 02847, 2020
iliekcomputers
Right
2020-01-28 02856, 2020
iliekcomputers
Here's what I got for now
2020-01-28 02857, 2020
sarthak_jain
Should I shoot a PR, so that you could check ?
2020-01-28 02803, 2020
iliekcomputers
We make dumps twice a month.
2020-01-28 02816, 2020
iliekcomputers
We should reload the data into the cluster twice a month too
2020-01-28 02827, 2020
iliekcomputers
From the full dumps
2020-01-28 02828, 2020
ruaok nods
2020-01-28 02839, 2020
ruaok
ideally load before running stats. :)
2020-01-28 02811, 2020
iliekcomputers
That way we can release stats while I look into what the best way to handle deleted users in the incremental dumps is.
2020-01-28 02836, 2020
ruaok
O_O
2020-01-28 02852, 2020
ruaok
I hadn't considered that wrinkle. but yes, that plan is good.
2020-01-28 02826, 2020
iliekcomputers
I realized the consistency problems in incremental dumps was probably due to someone deleting their account.
2020-01-28 02829, 2020
iliekcomputers
Ok
2020-01-28 02839, 2020
iliekcomputers
So action items for now are
2020-01-28 02856, 2020
iliekcomputers
Add cron jobs to the cluster which import new dumps
2020-01-28 02858, 2020
ruaok
I guess we need to add a "user deleted" message into the incrementals...
2020-01-28 02813, 2020
iliekcomputers
Add cron job to lemmy which requests stats
2020-01-28 02856, 2020
abhishekpanwar has quit
2020-01-28 02800, 2020
iliekcomputers
I'm not sure if the cluster needs to be "productionized" any more than just creating a new account from which to run prod services
2020-01-28 02829, 2020
ruaok
likely, but...
2020-01-28 02835, 2020
iliekcomputers
Once the cron jobs are added, we let it run for a month in beta and then do a release.
2020-01-28 02803, 2020
ruaok
a proper production cluster should run on a VLAN with a different firewall rules and the like.
2020-01-28 02821, 2020
iliekcomputers
Ah
2020-01-28 02829, 2020
ruaok
but, our cluster only really contains public data, so if that gets hacked, its not the end of the world.
2020-01-28 02842, 2020
ruaok
at this point I would suggest that we add cron jobs and move on.
2020-01-28 02801, 2020
iliekcomputers
Sure, makes life easier.
2020-01-28 02807, 2020
ruaok
I've asked zas to take a look at how to restrict the docker daemon to run on only one specific nic.
2020-01-28 02833, 2020
ruaok
that would allow me to easily setup a new cluster with more consistent hardware, rather than the hodgepodge I cobbled together a few weeks ago.
2020-01-28 02854, 2020
iliekcomputers
Other than that, I'm gonna look into observability a bit. If we could get notified somewhere that cron jobs are firing and working well, that'd be pretty nice.
2020-01-28 02833, 2020
ruaok
can we use sentry for that?
2020-01-28 02841, 2020
ruaok
I suppose that is an inverted use case.
2020-01-28 02823, 2020
iliekcomputers
I'll probably just add some code to send us an email
2020-01-28 02828, 2020
ruaok
but, I agree. we should be keeping a closer eye on things for the first few months.
2020-01-28 02834, 2020
iliekcomputers
Haven't thought about it much.
2020-01-28 02844, 2020
ruaok
ok, that is what we did with MB for... a decade?
2020-01-28 02801, 2020
iliekcomputers
Haha. Then, it works? :D
2020-01-28 02813, 2020
ruaok
if the humans read the email, yes.
2020-01-28 02831, 2020
iliekcomputers
Sure. I'll read em.
2020-01-28 02838, 2020
iliekcomputers sending mails to himself
2020-01-28 02804, 2020
ruaok
copy me. if we make it clear in the subject line if the thing failed, then it is much more likely get to get proper attention.
2020-01-28 02804, 2020
iliekcomputers
So that's the basic gist of the progress and the plan forward.
2020-01-28 02859, 2020
ruaok
sounds good. ping me if you need help with the setup of the cron jobs.
2020-01-28 02807, 2020
ruaok
I am not sure if the cluster is setup to email right now.
2020-01-28 02835, 2020
ruaok
zas: how difficult is it to get one machine to be able to send mail as meb?
2020-01-28 02844, 2020
iliekcomputers
Mhmm, will probably need help with that stuff.
2020-01-28 02811, 2020
ruaok
let me work with zas on getting email setup on the cluster.
2020-01-28 02833, 2020
zas
ruaok: you need to add IP to gmail to allow relay, and ofc set up a sender
2020-01-28 02853, 2020
ruaok
ok, doesn't sound too bad.
2020-01-28 02808, 2020
ruaok
I've got it on my list and when I work on it later this week, I'll ping ya, ok?
2020-01-28 02809, 2020
iliekcomputers
ruaok: can you update trello with the tasks for cron and email
2020-01-28 02814, 2020
ruaok
k
2020-01-28 02855, 2020
iliekcomputers
Thanks!
2020-01-28 02801, 2020
ruaok
> Look into why number of listens in incremental dumps do not match up.
2020-01-28 02803, 2020
ruaok
that is done now?
2020-01-28 02812, 2020
iliekcomputers
I think so, yes.
2020-01-28 02847, 2020
iliekcomputers
Should be replaced with "allow for deletion of users/listens in incremental dumps"
2020-01-28 02852, 2020
c1e0_ has quit
2020-01-28 02815, 2020
zas
ruaok: about docker network thing, I planned to have a look tomorrow, though I'm not sure about your exact goals, I'll ping you to clarify
2020-01-28 02846, 2020
zas
I have to run off, gf 's birthday cannot be skipped
2020-01-28 02830, 2020
ruaok
zas:perfect, do that.
2020-01-28 02853, 2020
ruaok
iliekcomputers: done.
2020-01-28 02801, 2020
iliekcomputers
Thanks!
2020-01-28 02817, 2020
prabal has quit
2020-01-28 02811, 2020
reosarevok
yvanzo: around again, will check your comments :)
2020-01-28 02830, 2020
Freso
outsidecontext: MusicBottle used Weblate.
2020-01-28 02827, 2020
Mr_Monkey is always Webontime
2020-01-28 02811, 2020
reosarevok is neither weblate nor webearly, it's just webwhenhemeanstoarrive
2020-01-28 02830, 2020
ruaok
so, webspanish?
2020-01-28 02823, 2020
yvanzo
reosarevok: btw, you should be able to update beta now.
2020-01-28 02817, 2020
reosarevok
yvanzo: as in, we should do a beta release now?
2020-01-28 02833, 2020
CatQuest
zas: re transaltinos. right now i cant really get transifex to even *work* so a move t oa differnt platform would *add* my ability to translate (not that norwegian translation is super useful ,but)
2020-01-28 02835, 2020
sarthak_jain has quit
2020-01-28 02826, 2020
yvanzo
reosarevok: iirc, we should update beta as soon as possible, except when it’s frozen.
2020-01-28 02800, 2020
reosarevok
Ok
2020-01-28 02807, 2020
reosarevok
Is there anything that can be merged already, I wonder
CatQuest: How far can you get? It is possible to download strings to translate from Transifex too, work on them locally, and upload them to Transifex. That might be an option if you want to translate but can’t use Transifex’s translation UI
2020-01-28 02852, 2020
yvanzo
started to review that one already but it's big indeed!
2020-01-28 02829, 2020
Freso
bitmap: Ping? Did you see my question regarding your last message to me?
Freso: You have to run that on the database before pg_upgrade (so probably need to start it back up with the old version of pg)
2020-01-28 02809, 2020
yvanzo
reosarevok: IIUYC, it prevents data loss but it doesn’t solve a less-severe related issue about pregap?
2020-01-28 02819, 2020
reosarevok
What's the less severe issue?
2020-01-28 02850, 2020
reosarevok
I mean, I don't think being able to move from track 0 to track 1 when there's no discID is an issue
2020-01-28 02805, 2020
reosarevok
It's just the user can do something wrong
2020-01-28 02824, 2020
reosarevok
The user can also add a data track as a normal track and that will also block adding a discID, but that's because they're doing it wrong :)
2020-01-28 02833, 2020
yvanzo
Ok, and +1 for the comments.
2020-01-28 02833, 2020
reosarevok
"If possible, help message should give legitimate use cases for these risky options."
2020-01-28 02854, 2020
reosarevok
You mean "you should only uncheck the pregap checkbox if the disc doesn't have a pregap track"?
2020-01-28 02833, 2020
yvanzo
Maybe it would be possible to make it easier to attach a DiscID to tracklist with erroneous pregap/data track. (about users doing it wrong) (but that is a separate issue)
(But it becomes difficult to follow three topics at the same time here...)
2020-01-28 02805, 2020
CatQuest
genre's have mbids? :O
2020-01-28 02813, 2020
reosarevok
`No MusicBrainz entities match the MBID {mbid}. Either it’s incorrect, it was for an entity that has since been deleted, or it is for something else than an entity (for example, a relationship type).`,
2020-01-28 02822, 2020
yvanzo
I had the same reaction at first :O
2020-01-28 02836, 2020
CatQuest
erhmmmmm reosarevok I can "remove " it
2020-01-28 02841, 2020
reosarevok
yvanzo, you reviewed the code that meant they had MBIDs :p
2020-01-28 02802, 2020
yvanzo
reosarevok: please reply in the PR, I cannot follow all conversations at the same time here :D
