#metabrainz

www.jokeexplainer.com/2017/04/a-physicist-engin... :D

Now was ffs

2021-04-29 00:01:15,126: (psycopg2.OperationalError) could not write to file "base/pgsql_tmp/pgsql_tmp50662.0": No space left on device

[listenbrainz-server] jdaok opened pull request #1417 (master…importer-privacy-warning): LB-802: Importer Last.FM privacy warning https://github.com/metabrainz/listenbrainz-serv...

yvanzo, it would be nice if the script checked how much space was free and you should have a good estimate of how much each process will take, build, populating the database, building the index, and whether the operation is likley to succeed or not.

[listenbrainz-server] jdaok closed pull request #1251 (master…LB-802): LB-802: Last.fm importer user friendliness improvements https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] jdaok opened pull request #1418 (master…importer-progress-bar): Importer Progress Bar https://github.com/metabrainz/listenbrainz-serv...

ruaok: i looked into spark missing data. it seems incremental dumps have not been happening since 22, no dump -> no spark import. hence the missing data.

Jeez. Found out JetBrains software doesn't clean after itself, so I had 6 GB of old cache crap

Agghhh...over 4 hours into loading the indexes and lose power...of course my UPS is bad and I've been waiting for an opportunity to swap it out. Looks like I got one :/

:|

Now I don't know if my db will be corrupt, but I expect pgsql will check its tables when it comes up?

Mr_Monkey: regarding the react context PR, I had taken a shot at it the last month. I didn't get time to complete it. However, I was able to get the server side up and running at that time. https://github.com/metabrainz/listenbrainz-serv...

The frontend and backend parts are unrelated so we can merge them separately as well. If you prefer, I can add that to the current PR otherwise I can open a different PR.

Oh nice ! Let't keep them separate _lucifer, I've got another 20 modified files to commit on that one…

But that's great, We'll only have to connect the two after they're done

sure 👍

yup, that should be one or two line change

akshaaatt[m]: ping

Hi _lucifer !

akshaaatt[m], hi! i reviewed the tagger fix. just one change needed to add a user friendly message other than that its ready to merge and deploy.

Okayy I'm on it!

awesome, thanks!

So I'll add the toast in the catch block?

yeah that should work, just like the one that was already there

Cool

something like "Something went wrong! We could not select the directory"

Okaayy

Done _lucifer

The starting with tests PR is incomplete. We can review that at last and if possible I could work on it today as per your directions

[musicbrainz-android] amCap1712 merged pull request #65 (master…tagger-fix): Tagger Fix https://github.com/metabrainz/musicbrainz-andro...

i probably won't be able to get to that PR today. my intent is to merge the release PR next and deploy a new version.

Okaayyyy. Sounds nice!

i found few bugs in other parts of the codebase while reviewing the tagger PR

at some places we do not propogate failed api request errors to the UI so the UI keeps on showing loading screen

Yeah I saw that

Will work on it then.

you can see LookupRepositoryImpl for an example of how to do that

or CollectionRepository

both of those propogate errors, LoginRepository and TaggerRepository don't

Right. I can see it

moin!

_lucifer: yes, in fact the dump is broken again. `/code/listenbrainz/admin/create-dumps.sh: line 23: admin/config.sh: No such file or directory`

was the new startup stuff released?

no not yet, but alastairp was discussing some changes with atj yesterday about the startup stuff

just finishing these points this morning

but that error is even more surprising

ok, then something else is causing this.

that script is cursed

atj: not the scripts fault. :)

i manually looked around the files in the container earlier today and the file was present

at least the errors are reasonably helpful :)

_lucifer: indeed, that file is present.

I note from the error the include appears to be relative

and valid.

so is the including script in the correct working directory for that path to resolve?

when invoked inside the container it runs.

is it invoked by cron in the container?

yes.

is the wording directory set in the cron invocation?

s/wording/working

no, which seems to be the problem.

but the crontab didn't change, so something in the script when we moved the source includes must've borked it.

duh, found it.

https://github.com/metabrainz/listenbrainz-serv... ?

exactly.

:)

I'll redeploy the container once the incremental dump is done. then I'll invoke the import and stats.

[musicbrainz-android] amCap1712 merged pull request #66 (master…add-ons): Prepare for Release https://github.com/metabrainz/musicbrainz-andro...

Great!🥳

_lucifer: a new incremental dump is on the FTP site and I requested an incremental import, but no import happens. any ideas what might be going on?

duh. there is an incremental dump missing. shit.

yeah, i did see a request in the request consumer logs but nothing was downloaded

I wonder if the transmogrifier was faster if I rewrote it in C. I dislike how awfully slow it is. All that JSON serializing/deserializing.

whats the transmogrifier?

it takes the standard postgres formatted dump and converts it to parquet format for importing into spark.

pretty simple, really.

what if we moved this part to postgres assuming that's possible?

what if we moved this part to postgres assuming that's possible?

I dont think it is possible.

the PG dumps go directly to disk.

it might be faster if we fetched fro PG rather from disk, but I didn't want to put the strain of two complete dumps on the DB.

makes sense. i wonder if how much performance could be gained if it were done in C because ujson utilises does it all in C under the hood.

however, we could probably parallelize this part of code.

[musicbrainz-server] reosarevok opened pull request #2078 (master…MBS-11627): MBS-11627: Don’t show "delete" link in UserList for deleted editors https://github.com/metabrainz/musicbrainz-serve...

ruaok: what do you think say we read 8000 lines at a time from normal dump, send chunks of 1000 to 8 threads for serializing/deserializing then wait for their output and write it to the spark dump and repeat?

also, i just remembered let's apply for google oauth verification

_lucifer: yes, we could do that, but we have to remember that we're on a production server, so we can't impact regular traffic.

I'm inclined to not worry about it now. normally waiting for dumps is a non-issue.

let's apply for the oauth stuff after lunch, ok?

makes sense.

yeah sure.

Mr_Monkey: thoughts on https://github.com/metabrainz/listenbrainz-serv... ?

I'll follow your lead. We can do it in a separate PR to easy reviewing.

That's not an english sentence. "to ease reviewing."

But I agree we should hold off merging 1361 until the connect services page is done.

yup agreed, i have fixed the unit tests. I'll add some more and then get to the connect services page.

Happy to help on that one if you need it

thanks!

alastairp, have you had the chance to look at the spotify rewrite or youtube PRs specifically deleting the user on revoking authorization. I wanted to run a catch by you and see if we should change some things.

let me open it up, and you can run it past me now

cool. so the issue is we delete the user entry from external_service_oauth table but keep it in the listens_importer table if the user has externally revoked auth.

now to implement this, I added a `stop_import` flag https://github.com/metabrainz/listenbrainz-serv...

when we have to display the user's current connection status, we have to do something like this https://github.com/metabrainz/listenbrainz-serv...

we also have to add a bit of logic in the template to figure out the current status.

to delete the entry from the listens_import table, the only way is that the user unlinks manually.

now i wonder if this special logic is worth it, if instead of deleting the user we add a flag to listens_import table as you had suggested some time ago

all of this boils down to simply checking that flag.

right

so currently you check if there's an error by doing the outer join and seeing if there is a row in listens_importer, but no row in external_service_auth?

yeah right

ah, but get_user_import_details first selects from external_service_oauth

what does listens_importer contain? this is like a request of a user to import from a given service?

why don't you `SELECT FROM listens_import OUTER JOIN external_service_auth` ?

that'd make external_service_auth.whatever null in this case, which is almost as easy as a flag

oh interesting, that makes sense.

this is the listens importer table https://github.com/metabrainz/listenbrainz-serv...

yeah, right. this is what we should use as the list of services that a user has configured

regarding the connect/delete code - I was testing this yesterday on beta, so that looks pretty good

I'll try and have another detailed look through the auth changes later today

yeah the current thing works, i was wondering if it could be simplified

our university project finishes tomorrow, so theoretically we have to deliver a bunch of documents to deliver, and barely any of them are finished :-P

money runs out, so we have to be finished

so I'm a bit tearing-my-hair-out here too

sure, no hurry here.

but I'm aware that you're also waiting on me for this, so I'll try and get to it

sure, if you get time, that would be nice. otherwise we can do this later as well. thanks!

I hate yaml. :)

I like it personally, but it does include a lot of footguns.

"It snows a lot in False." :)

Yaml. Yuuuuck

_lucifer: lets do this oauth stuff.

does that mean we need to keep that stuff running on a server and not mess with it?

here's the guidelines https://support.google.com/cloud/answer/9110914...