ruaok: yes, all nodes are up and the cluster looks sane. however, unless we submit a request we cannot be sure it works as expected. i did not start the request consumer on the newleader because i have a few doubts about how we should do it. do we want to run it directly on newleader or in a container on it?
yvanzo, bitmap: we only have 4 tickets ready for beta. Do you think we can improve that today? :)
ruaok
mooin!
_lucifer: not sure in this case. what do you think?
_lucifer
ruaok, i do not know about the networking concerns in case we use docker for request consumer. if there aren't any, then maybe let's use docker because we can just bring down the image and start a new one when dependencies etc. change, running directly could make these change a bit more involved sometimes.
but we'll also need to figure how to make docker talk back and forth with the host. i don't know if that is easy or hard.
ruaok
I see no network concerns and I think you have a good point about pulling the container, so lets use docket.
for that, put the container in host mode -- in host mode the container runs inside the normal port mapping of the host.
and the container has direct access to the host network layer.
ruaok: can you review the above PR? i'll build and test the request consumer then. i haven't changed memory configurations yet. will do that once request consumer works.
alastairp
_lucifer: what do you mean it wraps both queues?
ruaok
_lucifer: will try soon. we're in the office and have heaps on.
alastairp
we were just talking about this, we probably need to re-address this PR, and check for null characters in the input in the webserver, so that we can return http400 to the user
because if we have webserver -> push to queue, and then a separate process that reads from queue -> push to database, it's the wrong place to catch the error because now it's disconnected from the user request
however I'm still really confused as to how we managed to catch a postgres error in the webserver and stop these errors from happening
_lucifer
I meant that _send_listens_to_queue checks and send the listens to the appropriate queue. oh ok! i understand now what you meant
ruaok: right, that looks OK. I was just unsure about the .rollback()
this connection exists only for a batch of listens, right?
not for many batches
question: this batch will have many listens from many people. if it fails, what happens?
all listens fail to insert, or just the one with \0 ?
ruaok
all fail.
because this is not the place for us to be validating listens.
that should happen earlier.
alastairp
yeah, sure
but does this mean that we will loose other users' listens with this fix?
ruaok
_lucifer: on 1384 how do you plan to test this migration? run the new music services tables script but not the migrate existing users script?
_lucifer
ruaok, i think we can run both as the migrate existing users is just copy existing users.
on test.lb it'll read the users from the new table and on prod from the old table
ruaok
and then truncate the table before the actual release of this into produtction?
_lucifer
yes
ruaok
ok, makes sense.
aight, given the current context and release plan, this is fine by me.
_lucifer
awesome, thanks!
while i am at it, do you have any PRs you want to include in the release?
Mr_Monkey: ^
ruaok
the ones that are critical are merged already.
(user similarity page)
alastairp
oh, you know what I think caused this
ruaok
reading 1388 right now... I see you're creating newcluster files in an effort to not overwrite the existing cluster setup for now, but that we'll get rid of these when we ditch the old cluster?
alastairp
I'm having flashbacks to the id3 spec
sumedh joined the channel
_lucifer
yes
we have to configure the test setups as well before removing the old files
so i created new ones, we can cleanup once the prod and test up are running well
alastairp
tests running
ruaok
ok, lgtm for testing purposes.
alastairp
there is a VW car called the id3
and so now it's difficult to search for 'id3 spec'