> cool, i'll need to make changes to spark side anyways to send just 1 message to keep stuff simpler on lemmy side.
2021-05-24 14411, 2021
ruaok
that is a HUGE amount of data to send in one message, I don't think that is a good idea. I would leave it as is for now. but for inserts, write 500 stats in one go.
2021-05-24 14416, 2021
ruaok
something configurable, at least.
2021-05-24 14456, 2021
_lucifer
yeah, right, the size would be too large. so let's modify spark so that each message has 500 stats or whatever the config is.
2021-05-24 14412, 2021
ruaok
on the gist, there is a tricky detail... indexes are not renamed after a table rename. so you need to rename them as well, which is an extra step. OR you can just give these indexes a unique name e.g. col1_col2_ndx_table_<timestamp> , where timestamp is just an epoch int time.
2021-05-24 14423, 2021
ruaok
this way you'll never have to rename indexes.
2021-05-24 14438, 2021
ruaok
> yeah, right, the size would be too large. so let's modify spark so that each message has 500 stats or whatever the config is.
2021-05-24 14440, 2021
ruaok
that works.
2021-05-24 14459, 2021
_lucifer
another issue is how do we know that all messages for stats have ended so that we know it is time to rename tables?
2021-05-24 14417, 2021
ruaok
pass a message with zero entries?
2021-05-24 14448, 2021
_lucifer
right, like a sentinel. that would probably work.
2021-05-24 14403, 2021
ruaok
yep
2021-05-24 14433, 2021
_lucifer
all stats are however stored in the same table, so we need the sentinel after the last stats type only.
2021-05-24 14420, 2021
ruaok
well, we will need to replace the WHOLE table for all stats types in one go.
2021-05-24 14448, 2021
ruaok
perhaps we need a "stats are a comin'" message and "stats be done" message.
2021-05-24 14452, 2021
_lucifer
right, so we need to wait for all stats to arrive.
2021-05-24 14407, 2021
_lucifer
yes that sounds better.
2021-05-24 14433, 2021
_lucifer
i think the rest of stuff should be similar to similar_users.
2021-05-24 14437, 2021
ruaok
do step 1 for the stats are coming. step 2 for the stats messages and step 3-6 for the stats are done message.
2021-05-24 14452, 2021
ruaok
that feels pretty right.
2021-05-24 14416, 2021
_lucifer
yes, much better than what we have currently.
2021-05-24 14454, 2021
_lucifer
thanks! i'll put all this in a ticket and assign to myself.
Hi! I have been planning to maintain a google doc https://bit.ly/3wIkukP where I will update my progress and add a few reminders/comments/anything of importance to it so that I am through with my work. The link is editable to all and you can add any suggestions for me as well. You could star this doc in case you want daily updates from my end 😇 Other than that what all do you expect from me by the end of the first
2021-05-24 14453, 2021
akshaaatt[m]
evaluation _lucifer ? According to the proposal that would fill me with the Tagger implementation, codebase documentation for it and the UI updates.
2021-05-24 14446, 2021
_lucifer
akshaaatt[m]: the doc looks super nice. however, I would suggest you to make a post on community.metabrainz.org and keep posting updates there (at least for the user facing improvements). this way the rest of community can remain up to date on the status of the work and make suggestion as well.
2021-05-24 14436, 2021
akshaaatt[m]
Okaay! That sounds great. So I'll formulate and make a new topic on the forum soon.
2021-05-24 14427, 2021
_lucifer
sounds good. regarding updates, i think once a week is fine, if possible preferably in the weekly meeting we have on monday. in case you cannot make it, you can always send your review to Freso before the meeting.
2021-05-24 14426, 2021
_lucifer
If you prefer with a more frequent timeline, feel free to tell its fine with me. and you can reach out any time you are stuck somewhere or would like to discuss something etc.
2021-05-24 14423, 2021
akshaaatt[m]
Thank you _lucifer ! So today is Monday right?! I'll attend my first meeting today then. I guess it starts in an hour from now?
2021-05-24 14448, 2021
_lucifer
yes right. awesome! :D
2021-05-24 14417, 2021
TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Agenda (next meeting: 2021-05-24): Reviews, change of IRC network [ORG-49] (Freso)
2021-05-24 14447, 2021
_lucifer
ruaok: zeppelin setup on leader on port 9999. the UI looks a bit different but is similar to Jupyter Notebook. i connect to it as `ssh -L 9999:leader:9999 leader.listenbrainz.org`
