yvanzo, it would be nice if the script checked how much space was free and you should have a good estimate of how much each process will take, build, populating the database, building the index, and whether the operation is likley to succeed or not.
ruaok: i looked into spark missing data. it seems incremental dumps have not been happening since 22, no dump -> no spark import. hence the missing data.
2021-04-29 11939, 2021
reosarevok
Jeez. Found out JetBrains software doesn't clean after itself, so I had 6 GB of old cache crap
2021-04-29 11940, 2021
nelgin joined the channel
2021-04-29 11924, 2021
nelgin
Agghhh...over 4 hours into loading the indexes and lose power...of course my UPS is bad and I've been waiting for an opportunity to swap it out. Looks like I got one :/
2021-04-29 11946, 2021
tykling_
:|
2021-04-29 11917, 2021
nelgin
Now I don't know if my db will be corrupt, but I expect pgsql will check its tables when it comes up?
2021-04-29 11931, 2021
Darkloke joined the channel
2021-04-29 11906, 2021
_lucifer
Mr_Monkey: regarding the react context PR, I had taken a shot at it the last month. I didn't get time to complete it. However, I was able to get the server side up and running at that time. https://github.com/metabrainz/listenbrainz-server…
2021-04-29 11925, 2021
_lucifer
The frontend and backend parts are unrelated so we can merge them separately as well. If you prefer, I can add that to the current PR otherwise I can open a different PR.
2021-04-29 11952, 2021
Mr_Monkey
Oh nice ! Let't keep them separate _lucifer, I've got another 20 modified files to commit on that one…
2021-04-29 11929, 2021
Mr_Monkey
But that's great, We'll only have to connect the two after they're done
2021-04-29 11930, 2021
_lucifer
sure 👍
2021-04-29 11948, 2021
_lucifer
yup, that should be one or two line change
2021-04-29 11918, 2021
_lucifer
akshaaatt[m]: ping
2021-04-29 11910, 2021
akshaaatt[m]
Hi _lucifer !
2021-04-29 11959, 2021
_lucifer
akshaaatt[m], hi! i reviewed the tagger fix. just one change needed to add a user friendly message other than that its ready to merge and deploy.
2021-04-29 11918, 2021
akshaaatt[m]
Okayy I'm on it!
2021-04-29 11934, 2021
_lucifer
awesome, thanks!
2021-04-29 11931, 2021
akshaaatt[m]
So I'll add the toast in the catch block?
2021-04-29 11903, 2021
_lucifer
yeah that should work, just like the one that was already there
2021-04-29 11913, 2021
akshaaatt[m]
Cool
2021-04-29 11951, 2021
_lucifer
something like "Something went wrong! We could not select the directory"
2021-04-29 11923, 2021
akshaaatt[m]
Okaayy
2021-04-29 11944, 2021
akshaaatt[m]
Done _lucifer
2021-04-29 11923, 2021
akshaaatt[m]
The starting with tests PR is incomplete. We can review that at last and if possible I could work on it today as per your directions
ruaok: what do you think say we read 8000 lines at a time from normal dump, send chunks of 1000 to 8 threads for serializing/deserializing then wait for their output and write it to the spark dump and repeat?
2021-04-29 11945, 2021
_lucifer
also, i just remembered let's apply for google oauth verification
I'll follow your lead. We can do it in a separate PR to easy reviewing.
2021-04-29 11959, 2021
Mr_Monkey
That's not an english sentence. "to ease reviewing."
2021-04-29 11904, 2021
Mr_Monkey
But I agree we should hold off merging 1361 until the connect services page is done.
2021-04-29 11959, 2021
_lucifer
yup agreed, i have fixed the unit tests. I'll add some more and then get to the connect services page.
2021-04-29 11935, 2021
Mr_Monkey
Happy to help on that one if you need it
2021-04-29 11957, 2021
_lucifer
thanks!
2021-04-29 11956, 2021
_lucifer
alastairp, have you had the chance to look at the spotify rewrite or youtube PRs specifically deleting the user on revoking authorization. I wanted to run a catch by you and see if we should change some things.
2021-04-29 11930, 2021
alastairp
let me open it up, and you can run it past me now
2021-04-29 11927, 2021
_lucifer
cool. so the issue is we delete the user entry from external_service_oauth table but keep it in the listens_importer table if the user has externally revoked auth.
we also have to add a bit of logic in the template to figure out the current status.
2021-04-29 11910, 2021
_lucifer
to delete the entry from the listens_import table, the only way is that the user unlinks manually.
2021-04-29 11908, 2021
_lucifer
now i wonder if this special logic is worth it, if instead of deleting the user we add a flag to listens_import table as you had suggested some time ago
2021-04-29 11936, 2021
_lucifer
all of this boils down to simply checking that flag.
2021-04-29 11939, 2021
alastairp
right
2021-04-29 11903, 2021
alastairp
so currently you check if there's an error by doing the outer join and seeing if there is a row in listens_importer, but no row in external_service_auth?
2021-04-29 11911, 2021
_lucifer
yeah right
2021-04-29 11941, 2021
alastairp
ah, but get_user_import_details first selects from external_service_oauth
2021-04-29 11902, 2021
alastairp
what does listens_importer contain? this is like a request of a user to import from a given service?
2021-04-29 11931, 2021
alastairp
why don't you `SELECT FROM listens_import OUTER JOIN external_service_auth` ?
2021-04-29 11951, 2021
alastairp
that'd make external_service_auth.whatever null in this case, which is almost as easy as a flag
