[listenbrainz-server] paramsingh merged pull request #865 (master…implement-lovehate-table): Implement the database changes and SQL scripts for the lovehate table https://github.com/metabrainz/listenbrainz-server…
2020-05-19 14001, 2020
Mr_Monkey
🚂 Choo choo ! The GSOC train has left the station !
(sorry, it sounds like I'm just dumping more work on you)
2020-05-19 14044, 2020
shivam-kapila
Got it. Export the feedback
2020-05-19 14007, 2020
alastairp
some background: this is a legal requirement of the GDPR law, users should be able to download all of the data that we collect about them.
2020-05-19 14038, 2020
alastairp
we already have it for exporting listens. This functionality should be extended to include any other types of data that users add to the database
2020-05-19 14049, 2020
shivam-kapila
> (sorry, it sounds like I'm just dumping more work on you)
2020-05-19 14049, 2020
shivam-kapila
No no. Infact this is much needed. Thanks for pointing out
2020-05-19 14056, 2020
shivam-kapila
I will open a ticket.
2020-05-19 14029, 2020
alastairp
👍
2020-05-19 14037, 2020
shivam-kapila
Shall I create an epic love/hate so that its easy to manage tickets for this project?
2020-05-19 14042, 2020
iliekcomputers
shivam-kapila: yes
2020-05-19 14049, 2020
iliekcomputers
that would be very helpful
2020-05-19 14019, 2020
alastairp
iliekcomputers: does profile export just stream straight out of influx at the moment?
2020-05-19 14036, 2020
iliekcomputers
alastairp: yes
2020-05-19 14052, 2020
alastairp
with 2 different types of data, maybe we need to think about a differnt way of doing this
2020-05-19 14012, 2020
alastairp
we have similar requirements in AB, both for PII download and for dataset download
2020-05-19 14042, 2020
alastairp
there are 2 ways that we can approach it. One is to make a background processing system - log requests, build files, email users when it's ready, delete them after x days
2020-05-19 14050, 2020
alastairp
this is a lot of work and includes lots of moving parts
2020-05-19 14026, 2020
alastairp
currently in AB I'm planning on streaming zip files. Maybe this could be used in LB too. Not sure what happens when the data gets so big that it hits the http timeout though
this issue is triggered by mb cron prod on hip at least
2020-05-19 14018, 2020
bitmap
the hourly or daily script?
2020-05-19 14021, 2020
rdswift has quit
2020-05-19 14035, 2020
yvanzo
daily for sure
2020-05-19 14052, 2020
yvanzo
not in hourly.
2020-05-19 14026, 2020
bitmap
for a ton of reports it looks like
2020-05-19 14053, 2020
yvanzo
it's been triggered by sir as well.
2020-05-19 14033, 2020
bitmap
I mean we could try running some of those queries on pink (the read-only parts at least) but it might depend on the amount of concurrent access to trigger the error
2020-05-19 14035, 2020
yvanzo
(when resuming)
2020-05-19 14054, 2020
yvanzo
or we can just deploy to postgres-williams and run reports from test
2020-05-19 14053, 2020
yvanzo
anyway this issue is documented, it just hits shm limits.
2020-05-19 14006, 2020
bitmap
sure, but restarting the pg master requires downtime so it's best to be sure it works beforehand :)
2020-05-19 14056, 2020
yvanzo
right, do we go for trying it on slave or williams/test then?
