#metabrainz

reosarevok: oh. well. I guess I should read the forums first :)

I mean that uses the WS, dunno if yours queries the DB?

yeah, mine goes to the db directly

(my db)

So it's not a complete dupe, since there's a lot of stuff that can't even be gotten via the ws

Might still be worth mentioning it on that forum post :)

saifulbkhan: https://tickets.metabrainz.org/projects/AB/issu... is a good small ticket

AB-296: Bulk low-level get fails if one of the mbids doesn't exist in the database

sure

alastairp: I'm on it! unless you have some comments to add...

perhaps I'll add a bit more data other than just artists :)

saifulbkhan: nope, this one is pretty self-explanatory

ok! thx then!

let me find you an example URL which breaks it

alright

saifulbkhan: as an example, http://acousticbrainz.org/api/v1/low-level?reco... works

but http://acousticbrainz.org/api/v1/low-level?reco... doesn't

(see I changed the last letter of the URL, so that the ID doesn't exist)

if you load your own data into your local acousticbrainz server, you'll have different IDs

alastairp: and we want it to not fail and give us data for whichever IDs actually exist

samj1912/picard#61 (untagged-2648d7b3060570139fec - 40114de : Laurent Monin): The build passed.

Change view : https://github.com/samj1912/picard/compare/unta...

Build details : https://travis-ci.org/samj1912/picard/builds/20...

saifulbkhan: right

ruaok: are you planning to look at db-move again today? Also, what was the simple stuff you talked about earlier, I can get to it today

I'm feeling a bit under the weather. I'm going to lay down for a bit and see if I can get to it later.

a small thing... the listens page. the previous version (alpha) had better timestamps on it (x hours ago vs unix timestamp)

we need to bring back the better timestamps.

hello :3

hi SothoTalKer

good morning, Monday

ruaok: no rush :), I will work on the timestamps

Maybe I'll add the listen count to user page also while I'm at it, should be simple, hopefully.

SothoTalKer: hi :)

iliekcomputers: sounds simple, no?

Yeah, it does.

Should be a simple query to influx, right?

would you like to discover them first or should we talk about them now? :)

Lol

there are two considerations:

1. can influx do that? (it can)

2. how expensive is this operation?

might be fine to ask if you have 1000 listens.

what if you have 10k? 100k? 1M, 10M?

should you really count 10M rows every time the user page is loaded?

I think CatQuest would be angry with you.

+1

gcilou: you got out of bed. i got back from work :)

SothoTalKer: well I've already taken a calculus test and been at a meeting this morning so.. ;)

Caching in redis is my first solution to everything, but that won't work well because the listen count can change rapidly.

Or maybe we keep a listencount in redis for every user and update accordingly

?

iliekcomputers: good thinking.

gcilou: busy busy :-)

but, there is another train of thought that people are converging to....

if you have 13 listens, its nice to know when you have 14.

but when you have 138,849 listens, do you care when you have 138,850?

so, one way to do it is to not give exact figures or to clearly state when the figure was last updated.

Would be nice to know anyways, imo

of course, and we should provide that info.

invent a button on the userpage

the thing is programmers tend to want to do things very accurately, but often that accuracy doesn't really get consumed by the end user.

depending on how much work it is, you could run a daily/hourly job. and have a button that lets users update it in between, if they really neeed it

We can do ranges and as SothoTalKer said, add an option for exact values, seems like a good compromise.

you can listen to 480 3minute songs per day

that's about 175000 songs per year

nonstop continuous listening

People new to LB would still like to see a count updating 😃

so the 10M listens are a bit over the top ;)

SothoTalKer: what is the possible downside for doing a cron job?

iliekcomputers: and yes, of course the count needs to update.

users don't get live data

but, given that a song is 3-5 minutes long and a user might go their page a times times during a song, do we need to recompute the count each time?

SothoTalKer: what else?

i think you have something on your mind

i do

what if you have 1M users in the system.

and 10k daily users.

bold goals

if you recompute the totals once a day, you're going to compute 990,000 totals for naught.

but i see your point

is the listen count public?

I'm just using round numbers as example.

I think it should be since the data is public.

if a userpage is browsed, don't calculate it every time. just when the data is too old?

yep, that is common pattern that works.

what would be a reasonable update window given this situation?

Depends on how many listens the user has

does it?

If the user is brand new, maybe 15 minutes

iliekcomputers: I think you're conflating two seperate solutions.

and do per user update intervals make sense? how could you implement that?

(not that conflating two separate solutions is bad, mind you)

the downside of a userpage view based system is that you cannot have a 'top list' :D

SothoTalKer: I don't follow

who has the most listens? which song was listened to most?

ruaok: ranges based on the current number of listens we have cached, when the user page gets requested we check the number of listens in redis and when it was calculated, and then according to this, we update (or don't)

I think you're on the right track. how could you simplify your idea?

consider timeout values of data items stored in redis.

ruaok: please elaborate

when you stick a piece of data into redis, you can give it a time-to-live, right?

after that time, redis dumps it from the cache.

Oh, I did not realize that

so, restate your ideas using this concept.

bitmap: some of the trigger functions seem to be the same except for a name

here are some as an example https://gist.github.com/gentlecat/86f088372f6c9...

search_annotation_delete_1 and search_annotation_delete_3 are used for different tables, but I wonder if it can be just one

That makes it easy, we just calculate on some request and put it in redis with an expiry time based on how many listens the user has

On the next request we check redis and if there is no data then we calculate again, otherwise we use the old one.

Gentlecat: I don't see why not, to the indexer they are the same

iliekcomputers: yup, that is what I had in mind.

the only difference I see is in "direct" functions, which reference "id" row instead of "gid"

the other way

Nice :)

does sir use the gid for something?

not sure if it does

there's https://github.com/metabrainz/sir/blob/master/s...

but that's for generating the functions

presumably when it handles something from the delete queue, it sends some message to solr with the gid somewhere

here https://github.com/metabrainz/sir/blob/master/s...

that's the only place I see where it handles `delete` messages

I guess I'll send an email to mineo, this is strange

well, maybe it's only indexed by gid, so that's what delete_many needs

note that delete_1 and delete_3 only have the row id anyway, so doing an extra query for the gid would be redundant there when the indexer has to do it anyway

right