ruaok: yes, all nodes are up and the cluster looks sane. however, unless we submit a request we cannot be sure it works as expected. i did not start the request consumer on the newleader because i have a few doubts about how we should do it. do we want to run it directly on newleader or in a container on it?
yvanzo, bitmap: we only have 4 tickets ready for beta. Do you think we can improve that today? :)
2021-04-14 10429, 2021
ruaok
mooin!
2021-04-14 10440, 2021
ruaok
_lucifer: not sure in this case. what do you think?
2021-04-14 10403, 2021
_lucifer
ruaok, i do not know about the networking concerns in case we use docker for request consumer. if there aren't any, then maybe let's use docker because we can just bring down the image and start a new one when dependencies etc. change, running directly could make these change a bit more involved sometimes.
2021-04-14 10447, 2021
_lucifer
but we'll also need to figure how to make docker talk back and forth with the host. i don't know if that is easy or hard.
2021-04-14 10450, 2021
ruaok
I see no network concerns and I think you have a good point about pulling the container, so lets use docket.
2021-04-14 10429, 2021
ruaok
for that, put the container in host mode -- in host mode the container runs inside the normal port mapping of the host.
2021-04-14 10447, 2021
ruaok
and the container has direct access to the host network layer.
ruaok: can you review the above PR? i'll build and test the request consumer then. i haven't changed memory configurations yet. will do that once request consumer works.
2021-04-14 10432, 2021
alastairp
_lucifer: what do you mean it wraps both queues?
2021-04-14 10422, 2021
ruaok
_lucifer: will try soon. we're in the office and have heaps on.
2021-04-14 10442, 2021
alastairp
we were just talking about this, we probably need to re-address this PR, and check for null characters in the input in the webserver, so that we can return http400 to the user
2021-04-14 10422, 2021
alastairp
because if we have webserver -> push to queue, and then a separate process that reads from queue -> push to database, it's the wrong place to catch the error because now it's disconnected from the user request
2021-04-14 10441, 2021
alastairp
however I'm still really confused as to how we managed to catch a postgres error in the webserver and stop these errors from happening
2021-04-14 10443, 2021
_lucifer
I meant that _send_listens_to_queue checks and send the listens to the appropriate queue. oh ok! i understand now what you meant
ruaok: right, that looks OK. I was just unsure about the .rollback()
2021-04-14 10405, 2021
alastairp
this connection exists only for a batch of listens, right?
2021-04-14 10413, 2021
alastairp
not for many batches
2021-04-14 10433, 2021
alastairp
question: this batch will have many listens from many people. if it fails, what happens?
2021-04-14 10444, 2021
alastairp
all listens fail to insert, or just the one with \0 ?
2021-04-14 10400, 2021
ruaok
all fail.
2021-04-14 10420, 2021
ruaok
because this is not the place for us to be validating listens.
2021-04-14 10425, 2021
ruaok
that should happen earlier.
2021-04-14 10428, 2021
alastairp
yeah, sure
2021-04-14 10448, 2021
alastairp
but does this mean that we will loose other users' listens with this fix?
2021-04-14 10458, 2021
ruaok
_lucifer: on 1384 how do you plan to test this migration? run the new music services tables script but not the migrate existing users script?
2021-04-14 10408, 2021
_lucifer
ruaok, i think we can run both as the migrate existing users is just copy existing users.
2021-04-14 10434, 2021
_lucifer
on test.lb it'll read the users from the new table and on prod from the old table
2021-04-14 10436, 2021
ruaok
and then truncate the table before the actual release of this into produtction?
2021-04-14 10442, 2021
_lucifer
yes
2021-04-14 10446, 2021
ruaok
ok, makes sense.
2021-04-14 10448, 2021
ruaok
aight, given the current context and release plan, this is fine by me.
2021-04-14 10437, 2021
_lucifer
awesome, thanks!
2021-04-14 10416, 2021
_lucifer
while i am at it, do you have any PRs you want to include in the release?
2021-04-14 10419, 2021
_lucifer
Mr_Monkey: ^
2021-04-14 10440, 2021
ruaok
the ones that are critical are merged already.
2021-04-14 10447, 2021
ruaok
(user similarity page)
2021-04-14 10401, 2021
alastairp
oh, you know what I think caused this
2021-04-14 10404, 2021
ruaok
reading 1388 right now... I see you're creating newcluster files in an effort to not overwrite the existing cluster setup for now, but that we'll get rid of these when we ditch the old cluster?
2021-04-14 10409, 2021
alastairp
I'm having flashbacks to the id3 spec
2021-04-14 10411, 2021
sumedh joined the channel
2021-04-14 10428, 2021
_lucifer
yes
2021-04-14 10405, 2021
_lucifer
we have to configure the test setups as well before removing the old files
2021-04-14 10458, 2021
_lucifer
so i created new ones, we can cleanup once the prod and test up are running well
2021-04-14 10410, 2021
alastairp
tests running
2021-04-14 10410, 2021
ruaok
ok, lgtm for testing purposes.
2021-04-14 10423, 2021
alastairp
there is a VW car called the id3
2021-04-14 10430, 2021
alastairp
and so now it's difficult to search for 'id3 spec'
