#metabrainz

moooin!

look what was released ON THE DAY we upgraded to timescale 2.2.0: https://blog.timescale.com/blog/how-we-made-dis...

this exact release provides the feature we most needed. skip-indexes.

sigh. well, once we release things will be faster. and once we upgrade to 2.2.1 things will get even faster.

we can upgrade again, this one should be simpler

which provides a nice basis for us to scale the service.

_lucifer: agreed, but I think we should go pg12 as well this time.

but first I want to get time-ranges merged and deployed.

yeah makes sense to do that.

Mr_Monkey: we have a failing test for the frontend on the time-ranges PR. I'm hoping to merge and deploy this today: https://github.com/metabrainz/listenbrainz-serv...

can you please have a look?

that's the phantom bug Mr_Monkey mentioned yesterday methinks.

it goes away on upgrading node.

ruaok: amazing. glad to see that there's continual improvements going on with this work

_lucifer: so, we need to upgrade node in order to release this PR??

gah.

I'm just finishing some writing now. I've got a meeting at 12, so I'll start reviewing during it (sigh. mandatory meetings)

no, my understanding is that this issue only occurs in tests.

so we could mark the test as skipped for now and visit it again when we upgrade node.

if it works in prod I'm happy to merge despite a failing test

alastairp: ok, but lets really stay on point with this. listen counts are no longer being updated in production because of the crap disk space issues.

we really need to get this out today.

#1396 also needs to go out at the same time.

_lucifer: there is a failing spark test as well: https://github.com/metabrainz/listenbrainz-serv...

any ideas on that?

seems the build failed while trying to install python

i'll rerun the jobs and see if that goes away

ruaok: Yes, as mentionned I need to update Node version, but I started that last night.

Alternatively, you can merge as-is, I sexuple-checked that the mechanism works correctly, and using a more recent Node version all tests pass.

Mr_Monkey: great, thanks for verifying.

ruaok, i want to test some further improvements for the spark request consumer. will need to take down/up the container few times and issue some requests. is that fine?

Sure!

thanks!

[musicbrainz-server] reosarevok opened pull request #2094 (master…MBS-960): MBS-960: Allow mass-deleting and mass-changing user tags https://github.com/metabrainz/musicbrainz-serve...

ruaok, what's the right way to run a script in the background? i found nohup, setsid or & but not sure which to use?

script.py &

will run in the background, but if you close that terminal, the process gets killed.

nuhup script.py &

will continue running in the background.

nohup script.py 2>&1 > <file> &

should capture output into a file and keep running if the terminal disconnects.

yup that the last command is what i am exactly looking for. thanks!

likely to only work in bash.

alias m='make 2>&1 | less'

is an alias I have for make'ing C programs.

ruaok, driver running on newleader without docker now :D

and python deps also managed inside a venv so no issues there.

alastairp, after our discussion yesterday, i revisted the docs. i had misunderstood those, the deps do get distributed to the driver at least in the yarn client mode which is the one we use. thanks!

nice. remind me, what was the motivation for getting rid of docker?

doesn't add anything and requires additional setup. right we have to build the driver image and update the git repo on newleader to ensure things work correctly.

now its just update the repo on newleader and execute the startup script.

ruaok: I did a preliminary review of the PR, but skipped the listenstore file for now in order to get some comments to you. Looking at listenstore now

alastairp: great, thanks.

_lucifer: ahh, I see. I think we may want to make the request_consumer into an actual daemon, rather than starting with with nohup.

unit tests are passing in github, but failing for me (`self.listenstore.insert(listens)` -> AttributeError: 'NoneType' object has no attribute 'insert'), looking into why

I'll get moving on the other comments.

ruaok, sounds interesting. but i do not know how to do that. let's discuss it in detail later when you are available.

+1

_lucifer: in fact, the app startup process inside our docker (service files, etc) is exactly the same process as this, so we could reuse the mechanism

that would be even better :D

[listenbrainz-server] alastair opened pull request #1433 (master…spelling): Fix grammar https://github.com/metabrainz/listenbrainz-serv...

ruaok: do we have a small data dump that we can import for local development/testing?

no

ruaok: we can improve cache init in LB-879

LB-879: unify external service connections in single location https://tickets.metabrainz.org/browse/LB-879

k

me> wtf, why are all of my listens in the test database duplicated?

also me> oh that's right, I'm listening to my playlist of covers

Sigh, more fucking JIRA spam

I thought I blocked this asshole

I'm fixing it - lots of crap on AB

thanks reosarevok

alastairp: is https://tickets.metabrainz.org/browse/AB-409 meant to be fixed or not?

AB-409: Use bulk get methods for single recording endpoints

Freso: Bumping ROpdebee 's comment since it was easy to miss: This Jira user is spamming and possibly trying some SQL injection: https://tickets.metabrainz.org/secure/ViewProfi...

Oh, just seeing your comment reosarevok are you on this?

Mr_Monkey (and Freso): I blocked their jira editing privs (if I'm right about how that works) because just deleting the user makes it so that they can re-add it immediately

reosarevok: it looks like it's not fixed yet. leave open (or review submittedn) thanks

But this is the second time they do this

So, we could use a better way

They also tried to spam our github at some point

Thanks for dealing with it. Can't imagine what they're trying to get out of that.

I dont understand most spammers.

Chances are this specific one just has some sort of mental issue? The whole thing is super rambling. But who knows

Anyway, if you see it happen again, please do ping me, Freso or yvanzo ASAP

Any of us should be able to deal with it

Mr_Monkey: I'm seeing some weird frontend interaction issues with the new time ranges PR

after loading some listens from spotify. I'm looking into it a bit more, but I'm not sure if it's due to the time range storage/redis, or something in the frontend code

(local development version)

try deleting the redis volume and restarting.

trying that

first bits of feedback fixes pushed.

that doesn't seem to be it - I think it's mostly to do with the older/oldest newer/newest buttons. it's making the queries to get the updated data, but doesn't seem to be displaying it

Mr_Monkey: alastairp: https://vacunacovid.catsalut.gencat.cat/ is now starting to have useful info. Even though I fit the age bracket that is being vaccinated now, I tried for a cita and got a: Your group is hasn't been called yet. I'll keep trying.

I'm testing the api queries individually now

ruaok: cool, thanks (to both)

any comments on the listenstore yet?

if you have comments pending, post them please.

_lucifer: not that is sounds relevant anymore, but a proper linux daemon is created as such: https://stackoverflow.com/questions/17954432/cr...

thanks! i'll take a look.

ruaok: this is what I'm trying to work out with the frontend - now that I see some problems with the frontend I'll try and use the API to test the listenstore independently

what problems are you seeing? anything I can help with?

this is the data that I have in my local database

and redis

I haven't run anything to generate user counts in redis (lc.[username] key and lc-total? )

I started from an empty database and empty redis volume, created the db, logged in, and linked spotify

I dont see a problem so far.

listen_count_30day has a single row for me, says that count is 46. that looks good

when is the listenbrainz:lc.alastairp key created?

when get timestamps_per_user() is called.

this is from localhost. I don't know where (18611 total) is coming from here. coincidenally this is also the number displayed on listenbrainz.org: https://listenbrainz.org/user/alastairp

er no.

get_listen_count_for_user

all along the watchtower is nice song :)

there is an API call to fetch that.

I see it

/user/<user_name>/listen-count

call that and check the redis keys again

yep, the key is there now

you might need to configure the API_URL in config.py

ok, then it really is the front end.

by default it queries lb.org not the local

_lucifer: ohh, good one.

ahhhh - that'll be it. localhost dev will be hitting lb.org

that would do it.

thanks

phew.

> Recent listens (46 total)

phew

oh yeah, API works fine now

frontend does everything properly

Yay ! 🎉

_lucifer: I dont understand your last two questions on that PR "would this be... " and "I was thinking... "

well you and alastairp, really. is there an action for me?

i meant something on the line of HSET listenbrainz.ts:rob min 0 max 100 and then HGET listenbrainz.ts:rob min or HGETALL