i'm not sure what's happening. it worked when i manually triggered it
2020-09-04 24823, 2020
ishaanshah
Do you think we should move that part to python for better observabilty
2020-09-04 24828, 2020
iliekcomputers
yes, moving specific parts of it to python would probably be best.
2020-09-04 24843, 2020
iliekcomputers
i think we should do the cron changes that we're gonna do with incremental dumps
2020-09-04 24852, 2020
iliekcomputers
watch and see what happens
2020-09-04 24803, 2020
iliekcomputers
if it fails again, take a look
2020-09-04 24826, 2020
ishaanshah
Ohk,makes sense
2020-09-04 24856, 2020
ishaanshah
cool, so for the scheduling
2020-09-04 24808, 2020
ishaanshah
how much time does incremental dumps take to generate?
2020-09-04 24825, 2020
iliekcomputers
an hour, 2 max. but that's because we do it twice a week, the time would become lower if we do them everyday because the number of listens to dump decreases
2020-09-04 24838, 2020
iliekcomputers
let's start with one dump a day for now.
2020-09-04 24803, 2020
ishaanshah
Hmm, so maybe trigger a dump 6 PM UTC
2020-09-04 24820, 2020
ishaanshah
then trigger an import 9PM UTC
2020-09-04 24836, 2020
ishaanshah
and stats start at 00 UTC
2020-09-04 24805, 2020
iliekcomputers
mhmm, sounds reasonable to me.
2020-09-04 24808, 2020
ishaanshah
and a full dump each sunday
2020-09-04 24832, 2020
iliekcomputers
i'm tempted to make it twice a month
2020-09-04 24848, 2020
ishaanshah
Hmm,I am not too sure about that right now
2020-09-04 24858, 2020
iliekcomputers
why?
2020-09-04 24801, 2020
ishaanshah
We havent tested our incremental dump code yet
2020-09-04 24813, 2020
ishaanshah
One thing that worries me is the time taken to copy
2020-09-04 24821, 2020
ishaanshah
from /data/listenbrainz to /temp
2020-09-04 24802, 2020
ishaanshah
I think we should do atleast once a week for a month or so
2020-09-04 24813, 2020
iliekcomputers
the schedule is easy to change, i'd say let's go for twice a month and if we have problems we can bring it back.
2020-09-04 24831, 2020
abhinavohri joined the channel
2020-09-04 24841, 2020
ishaanshah
ok, so trigger the dump twice only?
2020-09-04 24853, 2020
kepstin has quit
2020-09-04 24854, 2020
maxlath[m] has quit
2020-09-04 24854, 2020
MagnusSvensson[m has quit
2020-09-04 24855, 2020
Leo_Verto has quit
2020-09-04 24857, 2020
joshuaboniface has quit
2020-09-04 24809, 2020
reg[m] has quit
2020-09-04 24811, 2020
goldenshimmer has quit
2020-09-04 24812, 2020
Cyna[m] has quit
2020-09-04 24814, 2020
iliekcomputers
sure
2020-09-04 24840, 2020
ishaanshah
cool, cool, I will incorporate all these things then
2020-09-04 24852, 2020
iliekcomputers
thanks!
2020-09-04 24829, 2020
ishaanshah
I wanted to talk to you about zeppelin integration with our spark cluster too
2020-09-04 24830, 2020
ishaanshah
I think it would be a nice improvement to the spark dev evironment
2020-09-04 24844, 2020
iliekcomputers
me too!
2020-09-04 24819, 2020
ishaanshah
Also, zeppelin has LDAP support, so we could provide access to the cluster to specific people
2020-09-04 24842, 2020
ishaanshah
like pristine___ can run her scripts safely in a sandbox
2020-09-04 24844, 2020
iliekcomputers
hmm, let's see how complicated it is to get into dev first, i'd say.
2020-09-04 24859, 2020
iliekcomputers
do you need help with anything specific yet?
2020-09-04 24808, 2020
ishaanshah
yep, I will start with the dev env first
2020-09-04 24834, 2020
ishaanshah
And if it looks good, lets look at setting it up for the cluster ig
2020-09-04 24852, 2020
iliekcomputers
awesome, i'm not very familiar with zeppelin specifically, but if you need any help, feel free to ping
2020-09-04 24804, 2020
ishaanshah
yep will do
2020-09-04 24817, 2020
ishaanshah
I will make the changes in the current PR and ping you then
MBS-10835: Disallow creating new accounts with an e-mail already in use
2020-09-04 24834, 2020
reosarevok
Last time a suggestion that seemed popular was to basically not set a verification email if the email can't be used
2020-09-04 24858, 2020
reosarevok
But that seems quite confusing, and I'd be the one dealing with the "my email never came!" complaints :p
2020-09-04 24826, 2020
iliekcomputers
ruaok: sorry to plague you with docs today but here's a wip version of the design doc. The user similarity storage story is still up in the air, but the rest of it is pretty straightforward https://www.dropbox.com/scl/fi/tcffmk725ylfx3melk…
2020-09-04 24809, 2020
ruaok
today was mostly earmarked for writing english, not code anyways. :)
2020-09-04 24821, 2020
kepstin joined the channel
2020-09-04 24821, 2020
iliekcomputers
:D
2020-09-04 24839, 2020
iliekcomputers
i've realized it's better to write english early in docs than late in PRs
2020-09-04 24851, 2020
iliekcomputers
ishaanshah: hey, one small thing, let's do the cron changes in a different PR
2020-09-04 24803, 2020
ruaok
yep, always good.
2020-09-04 24816, 2020
iliekcomputers
that way we can merge the one that's open, test out an incremental dump import and then change cron
2020-09-04 24817, 2020
reosarevok
yvanzo, bitmap, Freso, alastairp: Should we send an email saying "you tried to verify this email for musicbrainz account X, but it is already in use, please go to /account/edit and pick a different email to verify (or, if you had forgotten you already had an account, go to /lost-username to recover it!)"?
2020-09-04 24837, 2020
ishaanshah
iliekcomputers: cool, makes sense
2020-09-04 24842, 2020
reosarevok
(and then we can also explain them that they should contact us if they're trying to use it for a bot)
2020-09-04 24800, 2020
yvanzo
reosarevok: I would no suggest to pick a different email, just to go to /lost-username.
2020-09-04 24807, 2020
reosarevok
I guess
2020-09-04 24851, 2020
_lucifer
iliekcomputers: just a reminder on the performance doc, its ready from my side. can have a discussion whenever you are available
2020-09-04 24826, 2020
iliekcomputers
_lucifer: ah cool! let's chat over the weekend or Monday.
... I guess $username and $new_account are the same, but eh :p
2020-09-04 24818, 2020
alastairp
reosarevok: "if you didn't make this reqest, please contact us at support@..."
2020-09-04 24829, 2020
reosarevok
I guess I could add that, sure
2020-09-04 24837, 2020
reosarevok
Or just "reply and let us know", really
2020-09-04 24853, 2020
alastairp
wow, an organisation who doesn't use noreply@ !
2020-09-04 24856, 2020
reosarevok
Not sure what we'd do with that info, of course, but
2020-09-04 24808, 2020
reosarevok
IIRC we do but the email reaches us anyway :p
2020-09-04 24839, 2020
reosarevok
Yeah
2020-09-04 24842, 2020
reosarevok
'From' => $EMAIL_NOREPLY_ADDRESS,
2020-09-04 24842, 2020
reosarevok
'Reply-To' => $EMAIL_SUPPORT_ADDRESS,
2020-09-04 24853, 2020
reosarevok
Because noreply never stopped anyone from trying to reply
2020-09-04 24801, 2020
reosarevok
(or well, at least it never stopped *everyone*)
2020-09-04 24839, 2020
alastairp
mostly for the case when someone is actively trying to take over an account (or email address), or probe for security vulnerabilities
2020-09-04 24853, 2020
reosarevok
Sure, I mean that I'm not sure we have a way to stop them from trying again
2020-09-04 24838, 2020
alastairp
but at least it gives us a bit of a heads-up, so that we can see if we should make any changes
2020-09-04 24826, 2020
reosarevok
I guess
2020-09-04 24839, 2020
reosarevok
(I see most big companies just say "if you didn't request this, ignore it")
2020-09-04 24804, 2020
reosarevok
Our normal verification email doesn't say anything about it at all, other than "This email was triggered by a request from the IP address [$ip]."
2020-09-04 24808, 2020
reosarevok
Maybe it should :)
2020-09-04 24809, 2020
alastairp
for user account/password related things, it's a bit more important thatn that, I think
2020-09-04 24824, 2020
reosarevok
I meant specifically for stuff like that
2020-09-04 24829, 2020
alastairp
I've had someone take over my amazon account before. I didn't know until they sent me an email saying "the email of your account has been changed to x"
2020-09-04 24835, 2020
reosarevok
"Someone is trying to reset the password for this account! if this wasn't you, ignore!"
2020-09-04 24847, 2020
reosarevok
(since it requires email confirmation)
2020-09-04 24850, 2020
reosarevok
But yeah
2020-09-04 24804, 2020
alastairp
I mean, google specifically says "if this wasn't you visit the security center"
2020-09-04 24820, 2020
reosarevok
Yeah, not saying nobody does better :)
2020-09-04 24822, 2020
reosarevok
And not saying we need to follow the lowest denominator
2020-09-04 24823, 2020
reosarevok
Just thinking that I've often seen "dude, whatever" emails about it :D
as an alternative: If you can guarantee 100% that an email sent to someone by a phisher or accidentally by entering the wrong email will _not_ cause a security breach, then feel free to add "ignore it" :)
2020-09-04 24823, 2020
alastairp
sounds nice to me
2020-09-04 24843, 2020
reosarevok
I guess the "If you didn't try to register a MusicBrainz account at all and you're not sure why you're receiving this, we'd be thankful if you replied to this email and let us know." bit could be added to the standard verification email too
2020-09-04 24849, 2020
alastairp
to confirm - the workflow is that you can create an account without verifying the email, but then you have to verify to be able to do some actions?
2020-09-04 24853, 2020
reosarevok
Yes
2020-09-04 24858, 2020
reosarevok
Like editing
2020-09-04 24805, 2020
reosarevok
And tagging, rating... most stuff really
2020-09-04 24810, 2020
alastairp
sure, sounds good to me then
2020-09-04 24820, 2020
reosarevok
Not sure if you can have, say, a LB account without an email. iliekcomputers?
2020-09-04 24800, 2020
iliekcomputers
If you have a MB account, you can have an LB account afaik
2020-09-04 24813, 2020
reosarevok
Sure, but does it require a verified email at all?
2020-09-04 24819, 2020
iliekcomputers
No
2020-09-04 24820, 2020
reosarevok
Ok
2020-09-04 24826, 2020
reosarevok
Not sure we care. ruaok, do we care?
2020-09-04 24838, 2020
iliekcomputers
Not yet, I guess
2020-09-04 24843, 2020
iliekcomputers
:D
2020-09-04 24828, 2020
ruaok agrees
2020-09-04 24838, 2020
ruaok
no wait.
2020-09-04 24839, 2020
ruaok
spammers.
2020-09-04 24815, 2020
ruaok
well, I suppose we can say: "If we think you're spamming, we'll send you a warning mail if you gave us an email. Without an email, we'll just block your account."
