I can move ahead for now with this discrepancy, generate the dumps and may be loot at it later
2020-02-17 04831, 2020
ruaok
I bet the nature of that tables makes the query more complex, no?
2020-02-17 04805, 2020
pristine__
Which tables?
2020-02-17 04811, 2020
ruaok
the ar-ar tables.
2020-02-17 04822, 2020
pristine__
The bidirectional nature? Yup
2020-02-17 04826, 2020
ruaok
can you please elaborate on the text at the bottom of the gist? I'm not following.
2020-02-17 04827, 2020
ruaok
yep.
2020-02-17 04858, 2020
iliekcomputers
man, this is hard to read when compared to sql
2020-02-17 04859, 2020
pristine__
Umm.... In a min ( getting some snacks for evening)
2020-02-17 04820, 2020
pristine__
iliekcomputers: it's about habit, maybe. I find it pretty cool now :p
2020-02-17 04851, 2020
ruaok
I'm with iliekcomputers. this is hard for me to get into this context.
2020-02-17 04852, 2020
iliekcomputers
🤷🏽♂️
2020-02-17 04852, 2020
iliekcomputers
the formatting isn't helpful tbh
2020-02-17 04834, 2020
ruaok
the part that is challenging for me is that I have never personally done a spark project.
2020-02-17 04810, 2020
ruaok
and not having done one doesn't give me the knowledge that allows me to easily step into the project.
2020-02-17 04807, 2020
pristine__
If there is a better way out, I'd love to know :)
2020-02-17 04805, 2020
iliekcomputers
ruaok: that was basically the reason for us using Spark SQL instead of the orm. none of us knew the spark API well, but all of us had good/okay sql knowledge.
2020-02-17 04836, 2020
ruaok nods at iliekcomputers
2020-02-17 04846, 2020
iliekcomputers
pristine__: i don't know what exactly the problem is, but I'd print out the actual rows we get that we don't expect.
2020-02-17 04847, 2020
ruaok
pristine__: improving the formatting would be a good start.
2020-02-17 04851, 2020
iliekcomputers
and see how they're getting there
2020-02-17 04822, 2020
iliekcomputers
hell, i'd probably write an equivalent sql query just to test it
TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | GSoC students start here: https://bit.ly/2NUaxgb | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Meeting agenda: Reviews, Google Code-in (Freso), VLC plugin (ruaok)
2020-02-17 04812, 2020
BestSteve has quit
2020-02-17 04843, 2020
BestSteve joined the channel
2020-02-17 04837, 2020
zas
outsidecontext: around?
2020-02-17 04804, 2020
outsidecontext
zas: a bit
2020-02-17 04844, 2020
zas
when do you want to proceed? we have the meeting in 2 hours, then I'll be away for diner
2020-02-17 04822, 2020
zas
we can do the release after (in 4 hours or so?)
2020-02-17 04856, 2020
outsidecontext
zas: if this would be ok for you it would be best. I'm currently quite busy and away later, I'll be available at 20:30 or so
2020-02-17 04825, 2020
zas
ok ping me when around, later is better for me too
2020-02-17 04825, 2020
outsidecontext
ok, thanks. I had also already prepared a blog post a while back. it is more detailed then usual, with screenshots and such. could you look over it?
it needs some final updating, I already wrote that a while back.
2020-02-17 04843, 2020
zas
429 respcodes are showing when rate limiting is triggered
2020-02-17 04802, 2020
ruaok
yeah. and if the caller was respecting the rate limits we should see nearly no 429s. but they are clearly there, so the caller is being naughty.
2020-02-17 04837, 2020
ruaok
and that nothing actually ends up in the DB, I have to wonder wtf is going on.
2020-02-17 04856, 2020
zas
when it happens backend is handling ~800 reqs in 2 minutes, that's not that much: 800/120 -> 6.66 req/s (that's a good number, for a doom metal fan like me)
2020-02-17 04817, 2020
ruaok
but not for the scalabilty of a web site it is not.
2020-02-17 04847, 2020
zas
did you inspect data in POST requests?
2020-02-17 04805, 2020
ruaok
and I have an idea why that is happening.. we spend too much time inspecting POST requests.
2020-02-17 04819, 2020
zas
k
2020-02-17 04823, 2020
ruaok
our code is nice and clean, but that causes the JSON to be parsed and serialized more than once. :(
