0:00 AM
ruaok
loads of 504s on MB. bitmap you about?
2021-03-12 07141, 2021
0:00 AM
bitmap
yeah, I'll check
2021-03-12 07147, 2021
0:00 AM
ruaok
thx
2021-03-12 07106, 2021
0:01 AM
ruaok
looks similar to what was going on a few days ago. yesterday?
2021-03-12 07119, 2021
0:03 AM
zas
Floyd under heavy load
2021-03-12 07145, 2021
0:06 AM
zas
Extra traffic on mb website
2021-03-12 07112, 2021
0:07 AM
bitmap
there are a lot of queries fetching areas' artists listings
2021-03-12 07126, 2021
0:16 AM
bitmap
zas: did you block any of the traffic? I'm seeing a lot less of those now
2021-03-12 07142, 2021
0:16 AM
zas
nope, I didn't
2021-03-12 07107, 2021
0:17 AM
bitmap
hrm, okay
2021-03-12 07113, 2021
0:17 AM
zas
is there a pattern in what you are seeing?
2021-03-12 07126, 2021
0:18 AM
bitmap
I was seeing quite a few requests of the form /area/<mbid>/artists on kiki, and the queries for those flooding floyd
2021-03-12 07139, 2021
0:18 AM
bitmap
now there are almost none
2021-03-12 07113, 2021
0:20 AM
bitmap
and the load average has dropped since they stopped
2021-03-12 07139, 2021
0:20 AM
zas
it seems very similar to MBS-11417, basically certain traffic on the website leads to huge load on floyd
2021-03-12 07141, 2021
0:20 AM
BrainzBot
2021-03-12 07112, 2021
0:21 AM
zas
the number of requests increased but not by a huge factor
2021-03-12 07132, 2021
0:21 AM
bitmap
yeah. I've been working on the artist page queries today, which are pretty bad -- I'll add the find_by_area for artists query to the list
2021-03-12 07130, 2021
0:22 AM
zas
also we could prolly improve caching
2021-03-12 07115, 2021
0:23 AM
bitmap
for sure
2021-03-12 07122, 2021
0:23 AM
zas
for now, I'll resume on what I was doing... sleeping ;)
2021-03-12 07140, 2021
0:23 AM
bitmap
though these lists of linked entities are hard to cache since they're basically like search results
2021-03-12 07150, 2021
0:23 AM
bitmap
materialized tables + indexes will help a lot with these
2021-03-12 07158, 2021
0:23 AM
bitmap
right, sleep well!
2021-03-12 07109, 2021
0:25 AM
zas
thanks, we have to improve rate limiting somehow
2021-03-12 07136, 2021
0:25 AM
zas
but in this case it seems there arent that many requests, that's more about the nature of those requests
2021-03-12 07146, 2021
1:07 AM
Lotheric
2021-03-12 07147, 2021
1:07 AM
BrainzBot
MBH-485: 502 Bad Gateway: Server Being Naughty page should be updated with new MusicBrainz logo
2021-03-12 07151, 2021
1:07 AM
Lotheric
^^ I think this is fixed
2021-03-12 07101, 2021
1:08 AM
Lotheric
I just got the new Naughty page :)
2021-03-12 07140, 2021
1:37 AM
davic has quit
2021-03-12 07156, 2021
4:41 AM
Dhanmoni joined the channel
2021-03-12 07115, 2021
4:42 AM
Dhanmoni has quit
2021-03-12 07143, 2021
5:31 AM
BrainzGit
2021-03-12 07143, 2021
5:59 AM
reosarevok
Lotheric: I think you might be right, feel free to close
2021-03-12 07135, 2021
6:07 AM
reosarevok
yvanzo: did you eventually unblock that big-ass release?
2021-03-12 07156, 2021
6:07 AM
reosarevok
If not, should we, or should we add a ticket to figure out what to do with those and leave it blocked in the meantime?
2021-03-12 07110, 2021
6:38 AM
Darkloke joined the channel
2021-03-12 07112, 2021
6:38 AM
yvanzo
reosarevok: not yet
2021-03-12 07135, 2021
6:40 AM
yvanzo
I looked at how to implement your idea of blocking resources after some timeouts.
2021-03-12 07110, 2021
7:12 AM
CatQuest
[21:18] <ruaok> I wonder if we should allow each team member to pick some open source projects
2021-03-12 07110, 2021
7:12 AM
CatQuest
oh oh oh I pick internet archive!
2021-03-12 07106, 2021
7:20 AM
reosarevok
I guess that's fair
2021-03-12 07145, 2021
7:20 AM
reosarevok
More relevant than the Estonian Nature Fond :D
2021-03-12 07140, 2021
7:32 AM
BrainzGit
2021-03-12 07113, 2021
7:34 AM
reosarevok
yvanzo, bitmap ^ would be nice to merge that for the release after this, since it includes a potential even if hugely unlikely private-collection-info leak
2021-03-12 07121, 2021
7:34 AM
reosarevok
Probably not worth rushing to do it for this release
2021-03-12 07153, 2021
8:02 AM
BrainzGit
2021-03-12 07155, 2021
8:16 AM
BrainzGit
2021-03-12 07151, 2021
8:37 AM
shivam-kapila
Good afternoon
2021-03-12 07111, 2021
8:38 AM
shivam-kapila
ruaok: did the latest commit work or not
2021-03-12 07141, 2021
8:38 AM
shivam-kapila
related> oh oh oh I pick internet archive!
2021-03-12 07142, 2021
8:38 AM
shivam-kapila
Nice pick :D
2021-03-12 07114, 2021
8:39 AM
shivam-kapila
Oh sorry didnt see merge conflicts thing
2021-03-12 07124, 2021
8:39 AM
shivam-kapila
I will resolve once I return
2021-03-12 07141, 2021
8:46 AM
Sigyn has quit
2021-03-12 07152, 2021
9:19 AM
BrainzGit
2021-03-12 07107, 2021
9:21 AM
ShivamAwasthi joined the channel
2021-03-12 07108, 2021
9:27 AM
BrainzGit
2021-03-12 07122, 2021
9:31 AM
ShivamAwasthi has quit
2021-03-12 07100, 2021
9:39 AM
ruaok
moooin!
2021-03-12 07104, 2021
9:39 AM
ruaok
shivam-kapila: no it did not.
2021-03-12 07151, 2021
9:46 AM
BrainzGit
2021-03-12 07111, 2021
10:09 AM
BrainzGit
2021-03-12 07118, 2021
10:17 AM
Zialus joined the channel
2021-03-12 07142, 2021
10:17 AM
Zialus_PT has quit
2021-03-12 07140, 2021
10:41 AM
Sigyn joined the channel
2021-03-12 07142, 2021
11:10 AM
BrainzGit
2021-03-12 07111, 2021
11:11 AM
BrainzGit
2021-03-12 07156, 2021
11:14 AM
iliekcomputers
good morning!
2021-03-12 07102, 2021
11:15 AM
ruaok
moin!
2021-03-12 07127, 2021
11:21 AM
BrainzGit
2021-03-12 07146, 2021
11:21 AM
CatQuest
ack, logged out of jira again!
2021-03-12 07146, 2021
11:21 AM
CatQuest
i logged in just a few days ago. i was fine a while but now it's back to logging me out often :/
2021-03-12 07113, 2021
11:22 AM
SothoTalKer joined the channel
2021-03-12 07155, 2021
11:23 AM
SothoTalKer_ has quit
2021-03-12 07118, 2021
11:25 AM
Mr_Monkey
ruaok, alastairp: I'm at OfficeBrainz, wanna have lunch juntos?
2021-03-12 07133, 2021
11:25 AM
alastairp
I'm in a meeting for a few more hours :(
2021-03-12 07136, 2021
11:25 AM
ruaok
sorry, got lunch plans. will be there after.
2021-03-12 07113, 2021
11:26 AM
Mr_Monkey
Okidoke !
2021-03-12 07126, 2021
11:26 AM
travis-ci joined the channel
2021-03-12 07126, 2021
11:26 AM
travis-ci
2021-03-12 07126, 2021
11:26 AM
travis-ci has left the channel
2021-03-12 07140, 2021
11:40 AM
MajorLurker joined the channel
2021-03-12 07143, 2021
11:54 AM
bouhormq joined the channel
2021-03-12 07158, 2021
11:57 AM
MajorLurker has quit
2021-03-12 07133, 2021
12:01 PM
bouhormq has quit
2021-03-12 07128, 2021
12:07 PM
shivam-kapila
_lucifer: I had raised an issue on spotipy. They made a PR to fix it
2021-03-12 07131, 2021
12:07 PM
shivam-kapila
2021-03-12 07140, 2021
12:07 PM
shivam-kapila
Reg: invalid grant
2021-03-12 07156, 2021
12:20 PM
alastairp
thanks for reporting that, shivam-kapila!
2021-03-12 07145, 2021
12:21 PM
shivam-kapila
Welcome :)
2021-03-12 07128, 2021
12:25 PM
dpmittal joined the channel
2021-03-12 07153, 2021
12:43 PM
_lucifer
shivam-kapila, that's nice :D, thanks!
2021-03-12 07122, 2021
13:15 PM
bouhormq joined the channel
2021-03-12 07124, 2021
13:15 PM
Darkloke has quit
2021-03-12 07148, 2021
13:22 PM
pristine___
ruaok: _lucifer any ideas on how we can go about calculating similar artists
2021-03-12 07154, 2021
13:22 PM
pristine___
In spark
2021-03-12 07142, 2021
13:23 PM
bouhormq has quit
2021-03-12 07131, 2021
13:25 PM
_lucifer
pristine___: as in collaborative filtering based on artists instead of recording?
2021-03-12 07135, 2021
13:26 PM
pristine___
Nope, in addition to recording recommendations, we want to calculate similar artists using spark
2021-03-12 07152, 2021
13:26 PM
pristine___
Not necessarily collaborative filtering
2021-03-12 07102, 2021
13:27 PM
pristine___
What did you use for user similarity?
2021-03-12 07113, 2021
13:27 PM
_lucifer
peasron correlation
2021-03-12 07114, 2021
13:28 PM
_lucifer
We create a matrix where users are columns and recordings are row, then use pearson correlation to find similarity between each pair of users
2021-03-12 07153, 2021
13:28 PM
_lucifer
(the values in the matrix are pplaycounts of that user for that recording)
2021-03-12 07117, 2021
13:29 PM
_lucifer
I don't think that would work for similar artists though
2021-03-12 07135, 2021
13:29 PM
_lucifer
how do we define two artists are similar?
2021-03-12 07106, 2021
13:30 PM
_lucifer
like for users we say that if two users listen to a recording similar number of times, they are similar
2021-03-12 07134, 2021
13:30 PM
_lucifer
alastairp might have some thoughts about this?
2021-03-12 07148, 2021
13:32 PM
ruaok
I was thinking of using CF to create not similar artists but artists that you might get like.
2021-03-12 07114, 2021
13:33 PM
ruaok
To do that, feed artist counts, not recording counts to CF.
2021-03-12 07124, 2021
13:33 PM
_lucifer
yeah that makes sense to me
2021-03-12 07145, 2021
13:33 PM
ruaok
Read the recordings, but add an extra step to calculate artist counts, rather than track counts.
2021-03-12 07100, 2021
13:34 PM
ruaok
I think that will scale better than the pearson coeff.
2021-03-12 07117, 2021
13:37 PM
ruaok
Lucifer: did you see the email re the last.fm key?
2021-03-12 07116, 2021
13:38 PM
_lucifer
taking a look at it now
2021-03-12 07134, 2021
13:46 PM
alastairp
oh, but that's nice to see that it appears to be due to a mistake on our part, and not due to our regular usage
2021-03-12 07137, 2021
13:46 PM
BrainzGit
2021-03-12 07145, 2021
13:50 PM
pristine___
ruaok: oh, I misunderstood, I remember you wanted to get similar artists from spark because the current similar artist dump that comes from Musicbrainz and is based on collaborations didn't look promising to you, I guess our near aim is to get artist recommendations, for that I have some idea on how to move forward
2021-03-12 07129, 2021
13:55 PM
pristine___
_lucifer: what made you use peasron correlation? From what I read, the same thing for user similarity is doeable using CF too, are there additional perks in pearson correlation?
2021-03-12 07132, 2021
13:56 PM
ruaok
pristine___: didn't we chat about all this before?
2021-03-12 07151, 2021
13:56 PM
ruaok
Yes, the goal is to replace the similar artists stuff.
2021-03-12 07109, 2021
13:57 PM
ruaok
Replacing it with artists you might like should work fine.
2021-03-12 07119, 2021
13:57 PM
ruaok
But we get more usable data out of it.
2021-03-12 07138, 2021
13:57 PM
_lucifer
pristine___: the CF in spark does not allow to get the similarity matrix, which we wanted.
2021-03-12 07106, 2021
13:58 PM
_lucifer
ruaok, i think they are correct. we retry continuously every 3s without any limit.
2021-03-12 07115, 2021
13:58 PM
_lucifer
2021-03-12 07156, 2021
13:58 PM
ruaok
Please make a pr soon, _lucifer . Thanks!
2021-03-12 07133, 2021
13:59 PM
_lucifer
i think we could limit it to say 3-5 times max? but, it seems we do it completely on the client side. so the limit will be per user.
2021-03-12 07146, 2021
13:59 PM
pristine___
ruaok: from what I remember we decided to do artist recommendations in near future because it is easier, don't really remember about replacing similar artists with artist you might like
2021-03-12 07148, 2021
14:35 PM
bouhormq joined the channel
2021-03-12 07121, 2021
14:37 PM
bouhormq has quit
2021-03-12 07102, 2021
14:53 PM
bouhormq joined the channel
2021-03-12 07149, 2021
14:53 PM
shivam-kapila
ruaok: I tried rebasing and merging master
2021-03-12 07155, 2021
14:53 PM
shivam-kapila
It says up to date
2021-03-12 07120, 2021
14:54 PM
shivam-kapila
which file is causing merge conflicts?
2021-03-12 07121, 2021
14:54 PM
bouhormq has quit
2021-03-12 07152, 2021
15:02 PM
_lucifer
Mr_Monkey: ping
2021-03-12 07158, 2021
15:02 PM
Mr_Monkey
Pong
2021-03-12 07151, 2021
15:03 PM
shivam-kapila
Latency: 0.5 sec
2021-03-12 07126, 2021
15:05 PM
_lucifer
2021-03-12 07133, 2021
15:07 PM
_lucifer
does this have something to do with `jest.runAllTimers`?
2021-03-12 07124, 2021
15:10 PM
Mr_Monkey
I'm not sure, to be honest. Looking at the code I have the same understanding as you do
2021-03-12 07151, 2021
15:12 PM
Mr_Monkey
And also, shouldn't `setTimeout` be replaced by a spy function?
2021-03-12 07107, 2021
15:13 PM
Mr_Monkey
I'm not sure these tests are properly implemented
2021-03-12 07154, 2021
15:13 PM
_lucifer
oh :(. i'll test manually first then. i need to read up on jest spies.
2021-03-12 07125, 2021
15:15 PM
Mr_Monkey
So I think the way to implement this specific test is to `.spyOn(instance, 'getPage')` and expect it to be called more than once
2021-03-12 07102, 2021
15:16 PM
Mr_Monkey
(once from calling `await instance.getPage(1);` in the test, and after that called by the retry function)
2021-03-12 07114, 2021
15:16 PM
Mr_Monkey
Give that a try?
