-
Gentlecat
right, that's why skipping items would be useful
2016-06-22 17415, 2016
-
alastairp
yeah, but I'm not sure we should do it as a
2016-06-22 17434, 2016
-
alastairp
[x] skip adding mbids which don't exist in the AB database
2016-06-22 17401, 2016
-
Gentlecat
seems fine to me
2016-06-22 17430, 2016
-
Gentlecat
if you are fine with some items being removed, here's an option
2016-06-22 17459, 2016
-
Gentlecat
also useful if you want to create a dataset and not worry about filtering out items that aren't in AB yet
2016-06-22 17423, 2016
-
alastairp
yep
2016-06-22 17412, 2016
-
Leftmost` is now known as Leftmost
2016-06-22 17420, 2016
-
Gentlecat
anyway, bicycle is not going to ride itself
2016-06-22 17427, 2016
-
Gentlecat is away
2016-06-22 17441, 2016
-
kwikadi_ joined the channel
2016-06-22 17445, 2016
-
djwhitey has quit
2016-06-22 17445, 2016
-
kwikadi has quit
2016-06-22 17453, 2016
-
kwikadi_ is now known as kwikadi
2016-06-22 17457, 2016
-
djwhitey joined the channel
2016-06-22 17406, 2016
-
armalcolite
ruaok: docker-compose is version 1.5.2 while version2 configuration requires at least version 1.6
2016-06-22 17401, 2016
-
armalcolite
ruaok: i tried upgrading it but it is not upgrading beyond it.
2016-06-22 17411, 2016
-
ruaok
uninstall both docker and docker-compose and then follow these instructions to install them:
2016-06-22 17412, 2016
-
ruaok
2016-06-22 17416, 2016
-
ruaok
(use the curl command)
2016-06-22 17436, 2016
-
ruaok
2016-06-22 17440, 2016
-
ruaok
curl here as well
2016-06-22 17408, 2016
-
Slurpee joined the channel
2016-06-22 17446, 2016
-
diana_olhovyk_ has quit
2016-06-22 17458, 2016
-
ruaok
zas: how is the FTP server image coming?
2016-06-22 17404, 2016
-
ruaok
we really really need to get that online?
2016-06-22 17422, 2016
-
ruaok
should we do a non docker deploy just to get things moving again?
2016-06-22 17414, 2016
-
MBJenkins
2016-06-22 17431, 2016
-
JonnyJD joined the channel
2016-06-22 17456, 2016
-
lks joined the channel
2016-06-22 17452, 2016
-
JesseW joined the channel
2016-06-22 17439, 2016
-
pingupingu joined the channel
2016-06-22 17459, 2016
-
pingupingu has quit
2016-06-22 17400, 2016
-
pingupingu joined the channel
2016-06-22 17425, 2016
-
Mineo joined the channel
2016-06-22 17410, 2016
-
armalcolite
ruaok: it got upgraded. now its fetching layers. again.
2016-06-22 17416, 2016
-
armalcolite
ruaok
2016-06-22 17428, 2016
-
armalcolite
ruaok: around 30% it downloaded.
2016-06-22 17424, 2016
-
alastairp
slow internet :(
2016-06-22 17436, 2016
-
alastairp
is there another place you can go that has better internet, just to download?
2016-06-22 17405, 2016
-
armalcolite
alastairp: actually my connection gives about 2-10 Mbps but that fetching layer is very slow.
2016-06-22 17409, 2016
-
armalcolite
i dont know why
2016-06-22 17419, 2016
-
alastairp
a problem specifically with that server
2016-06-22 17438, 2016
-
alastairp
do you know what it is downloading? perhaps you could change a mirror somehow?
2016-06-22 17403, 2016
-
armalcolite
python
2016-06-22 17404, 2016
-
armalcolite
2016-06-22 17402, 2016
-
alastairp
this is the docker branch?
2016-06-22 17405, 2016
-
alastairp plays
2016-06-22 17408, 2016
-
armalcolite
yes
2016-06-22 17448, 2016
-
alastairp
since ruaok merged in the postgres stuff, we will have to take a look at your import/export branches again
2016-06-22 17402, 2016
-
alastairp
perhaps it will be simple - just re-implement a get_all_listens() method?
2016-06-22 17435, 2016
-
armalcolite
wrapper?
2016-06-22 17445, 2016
-
alastairp
cassandra.fetch_listens(current_user.musicbrainz_id)
2016-06-22 17456, 2016
-
alastairp
actually, that should be a really simple change, right? :)
2016-06-22 17414, 2016
-
armalcolite
2016-06-22 17446, 2016
-
armalcolite
i implemented this in postgres branch, so now its in master
2016-06-22 17405, 2016
-
armalcolite
import & export branches still use the old functions.
2016-06-22 17415, 2016
-
colbydray joined the channel
2016-06-22 17403, 2016
-
alastairp
cool
2016-06-22 17411, 2016
-
alastairp
I'm setting it up now
2016-06-22 17447, 2016
-
armalcolite
alastairp: it seems docker fetches slowly from behind proxies.
2016-06-22 17405, 2016
-
armalcolite
i connected VPN and its giving much better speed
2016-06-22 17410, 2016
-
alastairp
cool
2016-06-22 17421, 2016
-
alastairp
OK, I have docker installed. what should I run now?
2016-06-22 17445, 2016
-
armalcolite
these are some commands that ruaok told me.
2016-06-22 17447, 2016
-
armalcolite
docker build .
2016-06-22 17404, 2016
-
armalcolite
docker-compose build && docker-compose up
2016-06-22 17424, 2016
-
armalcolite
(in the top listenbrainz directory)
2016-06-22 17410, 2016
-
yeeeargh joined the channel
2016-06-22 17415, 2016
-
kartikgupta0909 has quit
2016-06-22 17448, 2016
-
jcazevedo has quit
2016-06-22 17402, 2016
-
ruaok
the docker-without-kafka branch is actually a much better branch to play with.
2016-06-22 17415, 2016
-
ruaok
the redis consumer isn't done yet, debugging that as we speak.
2016-06-22 17433, 2016
-
Zastai joined the channel
2016-06-22 17432, 2016
-
Leo_Verto has quit
2016-06-22 17448, 2016
-
fqtw_ joined the channel
2016-06-22 17455, 2016
-
regagain joined the channel
2016-06-22 17439, 2016
-
fqtw has quit
2016-06-22 17425, 2016
-
alastairp
that's an addition on top of the docker branch, I assume?
2016-06-22 17433, 2016
-
ruaok
branch off branch.
2016-06-22 17437, 2016
-
alastairp
sure
2016-06-22 17459, 2016
-
ruaok
I just didn't want to deal with the cassandra config, since I've decded to rip it and kafka out.
2016-06-22 17400, 2016
-
alastairp
and the idea is that if you merge that into master, we'd have docker + redis + postgres?
2016-06-22 17406, 2016
-
ruaok
yes
2016-06-22 17416, 2016
-
ruaok
and no cassandra and no kafka.
2016-06-22 17422, 2016
-
alastairp
hmm
2016-06-22 17424, 2016
-
ruaok
ideally nothing based on java and java thinking.
2016-06-22 17446, 2016
-
ruaok wasted *another* day trying to get kafka to behave.
2016-06-22 17410, 2016
-
ruaok
redis on the other hand is predictable and lovely.
2016-06-22 17427, 2016
-
alastairp
so, you'll just have a queue
2016-06-22 17434, 2016
-
alastairp
push on from the api, pop off from the consumer?
2016-06-22 17444, 2016
-
ruaok
yerp.
2016-06-22 17455, 2016
-
alastairp
will it grow forever?
2016-06-22 17458, 2016
-
ruaok
RPOPLPUSH based safe queu implementation.
2016-06-22 17403, 2016
-
alastairp
e.g., when you pop off, does it go away?
2016-06-22 17405, 2016
-
alastairp
RFOLPOP
2016-06-22 17411, 2016
-
ruaok
yes.
2016-06-22 17412, 2016
-
alastairp
ROFLPOP
2016-06-22 17429, 2016
-
ruaok
the idea is the redis is only a buffer.
2016-06-22 17451, 2016
-
JesseW has quit
2016-06-22 17438, 2016
-
ruaok
when a "with self.engine.connect() as connection" block exits is the transaction committed or does an explicit commit still need to happen?
2016-06-22 17430, 2016
-
alastairp
it commits automatically when exiting the block
2016-06-22 17440, 2016
-
alastairp
this is the biggest reason we switched to it
2016-06-22 17443, 2016
-
ruaok
k, thx.
2016-06-22 17409, 2016
-
ruaok
the code currently has it to toss the whole transaction if there is one bad listen.
2016-06-22 17420, 2016
-
ruaok
I'll validate listens beforehand to avoid that.
2016-06-22 17432, 2016
-
alastairp
of the 1000 from kafka?
2016-06-22 17437, 2016
-
alastairp
or whatever batchsize is
2016-06-22 17459, 2016
-
ruaok
yeah.
2016-06-22 17405, 2016
-
ruaok
I've yet to see any get written to PG.
2016-06-22 17443, 2016
-
alastairp
ok. I ran docker-compose up and there's a whole bunch of output
2016-06-22 17448, 2016
-
alastairp
does that mean I'm running?
2016-06-22 17418, 2016
-
ruaok
depends.
2016-06-22 17436, 2016
-
ruaok will post a clean startup in a sec
2016-06-22 17440, 2016
-
alastairp
consumer_1 | ImportError: No module named webserver.redis_connection
2016-06-22 17440, 2016
-
alastairp
listenbrainzserver_consumer_1 exited with code 1
2016-06-22 17445, 2016
-
alastairp
I guess that's not working
2016-06-22 17407, 2016
-
alastairp
but redis, db, web dockers seem to be workin
2016-06-22 17409, 2016
-
ruaok
looks to be expected for the current code you have.
2016-06-22 17419, 2016
-
ruaok
once i get the consumer working, I'll push new code.
2016-06-22 17431, 2016
-
alastairp
cool. localhost:8000 works!
2016-06-22 17431, 2016
-
alastairp
nice
2016-06-22 17437, 2016
-
ruaok
:-D
2016-06-22 17451, 2016
-
alastairp
how do I connect to postgres?
2016-06-22 17458, 2016
-
ruaok
docker ps
2016-06-22 17405, 2016
-
ruaok
find the running postgres container
2016-06-22 17418, 2016
-
ruaok
docker exec -it <container id> bash
2016-06-22 17422, 2016
-
MajorLurker joined the channel
2016-06-22 17431, 2016
-
ruaok
psql -U listenbrainz -h db listenbrainz
2016-06-22 17455, 2016
-
ruaok
or instead of bash you can pass psql at the end of the docker command
2016-06-22 17436, 2016
-
armalcolite
i received the exact same error now as alastairp
2016-06-22 17445, 2016
-
kartikgupta0909 joined the channel
2016-06-22 17451, 2016
-
kartikgupta0909
alastairp: Hi!
2016-06-22 17459, 2016
-
alastairp
that's... interesting
2016-06-22 17422, 2016
-
alastairp
I guess there's a way of forwarding the port out to the host as well, so I can use psql on my machine?
2016-06-22 17424, 2016
-
armalcolite
but localhost does not work for me
2016-06-22 17448, 2016
-
armalcolite
listenbrainzserver_web_1 exited with code 1
2016-06-22 17421, 2016
-
CallerNo6 has quit
2016-06-22 17433, 2016
-
alastairp
armalcolite: what other output do you have for web_1 ?
2016-06-22 17451, 2016
-
armalcolite
kafka unavailable error
2016-06-22 17406, 2016
-
alastairp
right
2016-06-22 17412, 2016
-
alastairp
as ruaok said, try the other branch
2016-06-22 17424, 2016
-
armalcolite
2016-06-22 17430, 2016
-
armalcolite
ok.
2016-06-22 17447, 2016
-
MajorLurker has quit
2016-06-22 17438, 2016
-
alastairp
ruaok: so the postgres image is a public one?
2016-06-22 17447, 2016
-
ruaok
yep. so is redis.
2016-06-22 17452, 2016
-
ruaok
ideally we keep it like that.
2016-06-22 17453, 2016
-
alastairp
right
2016-06-22 17402, 2016
-
ruaok
all the tools we use from someone else are public images.
