17:50 PM
alastairp
what if, when we got a new model and needed to re-dump an old mbid, we just re-dump the entire file again
2017-10-24 29739, 2017
17:50 PM
alastairp
that way, when we uncompress over the top of an existing dump, it'll just overwrite the old one
2017-10-24 29707, 2017
17:51 PM
alastairp
actually, that's almost the same as saying "when we release a new model we make a new initial high-level dump"
2017-10-24 29746, 2017
17:51 PM
iliekcomputers
Yes. But how often are models released?
2017-10-24 29703, 2017
17:52 PM
alastairp
which I like - it removes a huge amount of complexity, and we don't plan to release new models very often
2017-10-24 29729, 2017
17:52 PM
alastairp
I don't think we'd make new model releases more than 2 or 3 times per year
2017-10-24 29710, 2017
17:53 PM
alastairp
it means we'd have to keep track of highlevel and lowlevel json dumps differently
2017-10-24 29722, 2017
17:53 PM
alastairp
(currently we use just one row in `incremental_dumps` for everything)
2017-10-24 29756, 2017
17:53 PM
iliekcomputers
That wouldn't be hard to do i guess.
2017-10-24 29714, 2017
17:54 PM
iliekcomputers
This sounds like a good solution to me.
2017-10-24 29754, 2017
17:54 PM
alastairp
me too - it sounds familiar
2017-10-24 29702, 2017
17:55 PM
alastairp
I think we came up with the same solution
2017-10-24 29709, 2017
17:56 PM
iliekcomputers
Lol :)
2017-10-24 29734, 2017
17:57 PM
iliekcomputers
alastairp: I'll open a few tickets and start working on them, I guess. We can supersede AB-97 with a new one saying no dump file should have more than X recordings.
2017-10-24 29734, 2017
17:57 PM
BrainzBot
2017-10-24 29747, 2017
17:57 PM
iliekcomputers
And one for the hl Json stuff
2017-10-24 29714, 2017
17:58 PM
alastairp
hold of on closing 97 for now
2017-10-24 29730, 2017
17:58 PM
iliekcomputers
Right, not closing it. Will just open a new one.
2017-10-24 29731, 2017
17:58 PM
alastairp
I'd like to see how big a 100k or 200k or 500k file is first
2017-10-24 29701, 2017
17:59 PM
alastairp
because I don't think we should go less than 100k in a dump-part
2017-10-24 29716, 2017
17:59 PM
alastairp
but if that's 5-10gb, then it's too big for a sample dump
2017-10-24 29749, 2017
17:59 PM
alastairp
(having said that, we should be able to make a sample dump of x size or n number using the dump script)
2017-10-24 29735, 2017
18:00 PM
jsturgis joined the channel
2017-10-24 29758, 2017
18:00 PM
iliekcomputers
Right.
2017-10-24 29741, 2017
18:01 PM
iliekcomputers
A sample dump would be at most 1-2 GB, I guess.
2017-10-24 29700, 2017
18:03 PM
alastairp
I think that the absolute maximum we should have is 1gb
2017-10-24 29741, 2017
18:04 PM
alastairp
maybe even less compressed - 2GB uncompressed hl and ll combined
2017-10-24 29742, 2017
18:04 PM
alastairp
or something
2017-10-24 29722, 2017
18:05 PM
alastairp
this is for people who might want to set up a dev environment to look at the server; or who want to play with the data for running experiments
2017-10-24 29743, 2017
18:05 PM
alastairp
they don't need lots, just enough to make a basic dataset, etc
2017-10-24 29731, 2017
18:06 PM
iliekcomputers
I get the use case, just having set up AB too :P
2017-10-24 29734, 2017
18:17 PM
iliekcomputers
AB-314, AB-315
2017-10-24 29734, 2017
18:17 PM
BrainzBot
2017-10-24 29734, 2017
18:17 PM
BrainzBot
2017-10-24 29738, 2017
18:34 PM
CatQuest
"D/T" it is then I guess
2017-10-24 29748, 2017
18:35 PM
CatQuest
ugh you know what is annoying?
2017-10-24 29713, 2017
18:36 PM
CatQuest
the "spam" measurments we have on the wiki not cikcing in until I'm half way trought typing a sentence
2017-10-24 29714, 2017
18:36 PM
Leftmost
If you say "Leftmost", that will be not-nice. :P
2017-10-24 29734, 2017
18:36 PM
iliekcom- has quit
2017-10-24 29735, 2017
18:36 PM
ferbncode has quit
2017-10-24 29716, 2017
18:37 PM
CatQuest
i am sure I'm more annoying thna leftmost anyway
2017-10-24 29730, 2017
18:37 PM
CatQuest
@I'm QuestMost
2017-10-24 29732, 2017
18:42 PM
alastairp
great, thanks iliekcomputers
2017-10-24 29718, 2017
19:02 PM
CatQuest
uhm, hey have you done anything to the stuff behind the tagging ability?
2017-10-24 29734, 2017
19:02 PM
CatQuest
i cant tag fro mthe sidebar anymore (the box in the "tags" tab still works)
2017-10-24 29722, 2017
19:03 PM
CatQuest
I could tag things just yesterday so..
2017-10-24 29747, 2017
19:08 PM
CatQuest
ListMyCDs: thanks for already finding this and making a bug ticket <3
2017-10-24 29724, 2017
19:14 PM
samj1912
2017-10-24 29735, 2017
19:14 PM
samj1912
me and zas are trying to dockerize this
2017-10-24 29749, 2017
19:14 PM
samj1912
zas: said you'd know about how to handle the pgsql plugins part
2017-10-24 29724, 2017
19:15 PM
bitmap
I thought it was already mostly dockerized
2017-10-24 29746, 2017
19:15 PM
samj1912
nope
2017-10-24 29755, 2017
19:15 PM
samj1912
Is it?
2017-10-24 29708, 2017
19:16 PM
samj1912
2017-10-24 29729, 2017
19:16 PM
bitmap
yes
2017-10-24 29744, 2017
19:16 PM
samj1912
but its missing components
2017-10-24 29744, 2017
19:16 PM
bitmap
what's missing?
2017-10-24 29747, 2017
19:16 PM
samj1912
mainly sir
2017-10-24 29755, 2017
19:16 PM
samj1912
and the indexing components
2017-10-24 29712, 2017
19:17 PM
samj1912
2017-10-24 29722, 2017
19:17 PM
samj1912
hmm
2017-10-24 29732, 2017
19:17 PM
agentsim_ has quit
2017-10-24 29748, 2017
19:17 PM
bitmap
right, Gentlecat had already deployed this stuff using docker, so I assume something is working
2017-10-24 29759, 2017
19:17 PM
agentsim joined the channel
2017-10-24 29743, 2017
19:18 PM
samj1912
zas is not aware of the current config/parts and neither am I (docker noob here)
2017-10-24 29759, 2017
19:18 PM
bitmap
that shouldn't be hard to figure out
2017-10-24 29741, 2017
19:19 PM
samj1912
I have listed out all the parts in the manual setup, can you help us figure out if they are already dockerized
2017-10-24 29707, 2017
19:20 PM
samj1912
zas: also pointed out a couple of mistakes with the existing config (data files in the same directory as config files)
2017-10-24 29725, 2017
19:20 PM
samj1912
and also we might want to run a solr cloud setup rather than the normal solr
2017-10-24 29752, 2017
19:22 PM
bitmap
2017-10-24 29701, 2017
19:23 PM
samj1912
okay
2017-10-24 29708, 2017
19:23 PM
bitmap
those are meant to be in the same container
2017-10-24 29734, 2017
19:23 PM
samj1912
okay
2017-10-24 29724, 2017
19:25 PM
samj1912
what about rabbitmq and connection to mbdb?
2017-10-24 29733, 2017
19:25 PM
bitmap
and afaiu mbdata and rngpy are just libraries used in the sir container
2017-10-24 29747, 2017
19:25 PM
samj1912
yes
2017-10-24 29751, 2017
19:25 PM
samj1912
they are installed with sir
2017-10-24 29709, 2017
19:27 PM
bitmap
2017-10-24 29709, 2017
19:28 PM
bitmap
2017-10-24 29743, 2017
19:28 PM
bitmap
it's a config template, rendererd by consul-template (which pulls that above data from consul)
2017-10-24 29717, 2017
19:29 PM
samj1912
cool and where do we install the pg_amqp plugin in the main db?
2017-10-24 29732, 2017
19:29 PM
samj1912
pointing to the above rabbitmq instance
2017-10-24 29736, 2017
19:29 PM
bitmap
that's already installed
2017-10-24 29706, 2017
19:30 PM
bitmap
2017-10-24 29730, 2017
19:30 PM
bitmap
but you'll need to insert a row into the amqp.broker table
2017-10-24 29746, 2017
19:30 PM
samj1912
okay, where will I have to do that?
2017-10-24 29759, 2017
19:30 PM
samj1912
2017-10-24 29713, 2017
19:31 PM
bitmap
2017-10-24 29740, 2017
19:31 PM
bitmap
has to be done manually currently (but it's just a one-time thing)
2017-10-24 29751, 2017
19:31 PM
samj1912
okay
2017-10-24 29710, 2017
19:32 PM
samj1912
the host remains the same always?
2017-10-24 29747, 2017
19:32 PM
bitmap
yeah, it's where rabbitmq is running, which is always on serge right now
2017-10-24 29710, 2017
19:33 PM
bitmap
there should already be a row in the broker table for the setup Gentlecat had, you can just use/modify that
2017-10-24 29723, 2017
19:33 PM
samj1912
okay
2017-10-24 29752, 2017
19:33 PM
samj1912
I think that takes care of everything
2017-10-24 29709, 2017
19:34 PM
samj1912
I will co-ordinate with zas to get a test setup via docker up and running
2017-10-24 29703, 2017
19:35 PM
samj1912
thanks for the help :D
2017-10-24 29728, 2017
19:35 PM
bitmap
sure, let me know if you need help with the amqp setup
2017-10-24 29748, 2017
19:35 PM
samj1912
yup
2017-10-24 29709, 2017
19:36 PM
samj1912
I haven't even looked into that part of SIR yet
2017-10-24 29725, 2017
19:36 PM
samj1912
will have to understand how it works first
2017-10-24 29717, 2017
19:37 PM
bitmap
if you see queries like 'PERFORM amqp.publish(2, ...)' in sir, the '2' is actually the row id in the broker table
2017-10-24 29734, 2017
19:37 PM
bitmap
that tells it where/how to connect to rabbitmq and where to publish things
2017-10-24 29747, 2017
19:37 PM
samj1912
okay
2017-10-24 29737, 2017
19:41 PM
samj1912
bitmap: one last thing, how do I add the triggers generated by SIR to MB DB?
2017-10-24 29743, 2017
19:44 PM
bitmap
cat sql/CreateFunctions.sql | psql -U musicbrainz db_name
2017-10-24 29758, 2017
19:44 PM
bitmap
cat sql/CreateTriggers.sql | psql -U musicbrainz db_name
2017-10-24 29702, 2017
19:45 PM
bitmap
something like that :)
2017-10-24 29714, 2017
19:45 PM
samj1912
okay, so I just run in manually once?
2017-10-24 29719, 2017
19:45 PM
bitmap
(those files are in the sir repo)
2017-10-24 29732, 2017
19:45 PM
bitmap
yes, unless the functions/triggers change and you need to update them
2017-10-24 29743, 2017
19:46 PM
samj1912
I haven't looked at the code yet, but I am not sure if the current triggers handle all the relationships and attributes I added over the last 2 weeks
2017-10-24 29704, 2017
19:50 PM
bitmap
you can regenerate them with the Makefile and see what changes
2017-10-24 29718, 2017
19:50 PM
bitmap
`make createsql` I guess
2017-10-24 29736, 2017
19:52 PM
github joined the channel
2017-10-24 29736, 2017
19:52 PM
github
[acousticbrainz-server] alastair closed pull request #235: Upgrade javascript dependencies (master...js-upgrade)
https://git.io/v5ht6
2017-10-24 29736, 2017
19:52 PM
github has left the channel
2017-10-24 29717, 2017
19:53 PM
bitmap
samj1912: are you setting this up on the production database (bowie)? or somewhere else
2017-10-24 29751, 2017
19:53 PM
bitmap
I mean if you're doing a test setup somewhere else first or something
2017-10-24 29701, 2017
19:54 PM
samj1912
I will first do it on a mbdb replicae
2017-10-24 29729, 2017
19:54 PM
samj1912
2017-10-24 29736, 2017
19:54 PM
samj1912
its a google cloud vm
2017-10-24 29702, 2017
19:55 PM
samj1912
2017-10-24 29718, 2017
19:56 PM
bitmap
okay, not sure how postgres is setup there, you may need to install pg_amqp by hand
2017-10-24 29727, 2017
19:56 PM
samj1912
okay
2017-10-24 29738, 2017
19:57 PM
bitmap
2017-10-24 29754, 2017
19:57 PM
samj1912
yup, installed it on my local setup
2017-10-24 29709, 2017
19:58 PM
samj1912
I have the entire thing running locally on solr-test
2017-10-24 29717, 2017
19:58 PM
samj1912
I just need to shift it to docker
2017-10-24 29723, 2017
20:08 PM
samj1912
bitmap: how do I call the GenerateSQLScript with correct arguments?
2017-10-24 29745, 2017
20:08 PM
samj1912
2017-10-24 29745, 2017
20:08 PM
github joined the channel
2017-10-24 29745, 2017
20:08 PM
github
2017-10-24 29745, 2017
20:08 PM
github has left the channel
2017-10-24 29702, 2017
20:09 PM
samj1912
because I can see a lot of missing statements in the dropsql part
2017-10-24 29727, 2017
20:09 PM
samj1912
I called it by `./GenerateSQLScripts.pl ../../sir/sql/ `
2017-10-24 29744, 2017
20:34 PM
agentsim_ joined the channel
2017-10-24 29742, 2017
20:38 PM
agentsim has quit
2017-10-24 29711, 2017
20:50 PM
samj1912
2017-10-24 29715, 2017
20:50 PM
samj1912
good night people o/
2017-10-24 29730, 2017
20:54 PM
aaa_ joined the channel
2017-10-24 29732, 2017
20:54 PM
aaa_ has quit
2017-10-24 29705, 2017
21:06 PM
CatQuest
natta samj1912 !
2017-10-24 29714, 2017
21:15 PM
Somasis is now known as squawk
2017-10-24 29721, 2017
21:15 PM
squawk is now known as Somasis
2017-10-24 29733, 2017
21:38 PM
bitmap
samj1912: it looks like it removed some stuff from the Drop* files that no longer exists, but nothing current is missing I don't think?
2017-10-24 29719, 2017
21:46 PM
kyan has quit
2017-10-24 29710, 2017
22:14 PM
bitmap
upgrading jenkins to fix some security vulnerabilities
2017-10-24 29737, 2017
23:22 PM
MajorLurker joined the channel
2017-10-24 29706, 2017
23:46 PM
ferbncode_ has quit
