I can move ahead for now with this discrepancy, generate the dumps and may be loot at it later
ruaok
I bet the nature of that tables makes the query more complex, no?
pristine__
Which tables?
ruaok
the ar-ar tables.
pristine__
The bidirectional nature? Yup
ruaok
can you please elaborate on the text at the bottom of the gist? I'm not following.
yep.
iliekcomputers
man, this is hard to read when compared to sql
pristine__
Umm.... In a min ( getting some snacks for evening)
iliekcomputers: it's about habit, maybe. I find it pretty cool now :p
ruaok
I'm with iliekcomputers. this is hard for me to get into this context.
iliekcomputers
🤷🏽♂️
the formatting isn't helpful tbh
ruaok
the part that is challenging for me is that I have never personally done a spark project.
and not having done one doesn't give me the knowledge that allows me to easily step into the project.
pristine__
If there is a better way out, I'd love to know :)
iliekcomputers
ruaok: that was basically the reason for us using Spark SQL instead of the orm. none of us knew the spark API well, but all of us had good/okay sql knowledge.
ruaok nods at iliekcomputers
pristine__: i don't know what exactly the problem is, but I'd print out the actual rows we get that we don't expect.
ruaok
pristine__: improving the formatting would be a good start.
iliekcomputers
and see how they're getting there
hell, i'd probably write an equivalent sql query just to test it
TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | GSoC students start here: https://bit.ly/2NUaxgb | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Meeting agenda: Reviews, Google Code-in (Freso), VLC plugin (ruaok)
BestSteve has quit
BestSteve joined the channel
zas
outsidecontext: around?
outsidecontext
zas: a bit
zas
when do you want to proceed? we have the meeting in 2 hours, then I'll be away for diner
we can do the release after (in 4 hours or so?)
outsidecontext
zas: if this would be ok for you it would be best. I'm currently quite busy and away later, I'll be available at 20:30 or so
zas
ok ping me when around, later is better for me too
outsidecontext
ok, thanks. I had also already prepared a blog post a while back. it is more detailed then usual, with screenshots and such. could you look over it?
it needs some final updating, I already wrote that a while back.
zas
429 respcodes are showing when rate limiting is triggered
ruaok
yeah. and if the caller was respecting the rate limits we should see nearly no 429s. but they are clearly there, so the caller is being naughty.
and that nothing actually ends up in the DB, I have to wonder wtf is going on.
zas
when it happens backend is handling ~800 reqs in 2 minutes, that's not that much: 800/120 -> 6.66 req/s (that's a good number, for a doom metal fan like me)
ruaok
but not for the scalabilty of a web site it is not.
zas
did you inspect data in POST requests?
ruaok
and I have an idea why that is happening.. we spend too much time inspecting POST requests.
zas
k
ruaok
our code is nice and clean, but that causes the JSON to be parsed and serialized more than once. :(