#metabrainz

https://musicbrainz.org/search?query=harpsicord... does find it

FishQuest: hmm, seems like dismax isn't one

*on

weird

try with https://musicbrainz.org/ws/2/instrument?query=h...

sorry

eehhh

?

dismax=true does nothing on test.mb

or is it url hack?

https://test.musicbrainz.org/search?query=harps... does nothing

ah the ~ 

uh

can't you give me an actaly search link?

ah https://test.musicbrainz.org/search?query=harps... works

yup

hmm, this was old functionality, when the indexed search first got turned on. it required things like ~ to be abl ot get similar results

i odn't remember whne it jsut beasme defualt

ugh

FishQuest: still testing things

i don't remember whne it just became defualt

sam <3 :)

search server isn't fully integrated with the ui yet

yep

you can see http://www.lucenetutorial.com/lucene-query-synt...

eh

for more cool stuff you can do

i thought this was the solr search?

yes, solr is built on lucene

that stuff.. I know that tuff been using it since we got lucene search :)

oh ho

this i didn't know, i thouht it wasa completely different thing

if lucene were bricks, solr is like a pre built house you can put your furniture in

the current search server we built from scratch

hmm

are you sure about that?

that we built it from scratch?

the way I remember is, that lucene was added and tinkered with tremendously, but it also ,came fro msomething already built

this was oh.. wtf 7 20 years ago?

erh 10 not 20

well, you get the point :P

anyway I'm going to the library <3, ping me when the test server can be logged into . (no rush or anything)

bitmap: ping me when you're caffeined enough

alastairp: the sharepoint download of all the files downloaded 20GB "successfully", but produces a corrupt zip file.

> 16455114579 extra bytes at beginning or within zipfile. zipfile corrupt.

ruaok: took from 10:48 to 13:!3 to index all recordings

oh wow. that is great.

2:25 hrs around

oh wait, there's more, it ended on 13:49 sorry so about 3 hrs

anything less than 6 hours is great. :)

ruaok: I have URLs to download with curl, but internet here is rate limited during the day

and zas pointed out that doing it over tcp has about 50-175% overhead depending on whether its ssl or not

hit me. I got 300mbit ready to go!

we figured we will move the slave to the same container and use sockets

zas is waiting for bitmap to figure out how to do it

and I dont think we have tuned the parameters enough yet

hopefully we should be able to get recording index down to 1 hr or 1.5 hrs

maybe less

me and zas were also discussing a ram only index if we want it really really quick in terms of indexing and retrieval, but it might be overkill :P since we have raid ssds

samj1912: don't worry about tuning too much.

ideally we will do this only once.

okay

alastairp: thanks. Now downloading MLHD at ~30MB/s. :)

incredible

datacenter to datacenter FTW

then I'll shove this into BigQuery.

we use google drive for the same reason to share stuff... enterprise file storage is way faster than the local internet connection

5 files done already.

I think it actually was a smart decision to put it on MS cloud, I guess McGill has an enterprise/academic account

ruaok: entire indexing done except editors and cdstubs

took exactly 4 hours for everything

but the whole point is to not reindex everything right ? how does it perform after one day of changes ?

ruaok: just looking at the contents of the tar archives... no subdirectories, individual files are gzip compressed

zas: not sure, haven't tested it yet

might be worth writing a quick script to uncompress the archives and put them on disk in a nice structure

(or upload to BQ directly from the tar??)

I need bitmap's help in adding the triggers

and setting up rabbitmq

ruaok: btw, Felipe suggested https://airflow.apache.org/ as a tool for managing data from a local datastore -> BQ

might be something that we could look at if we're planning on sending data from lots of places

I've not looked at it yet, but I'm going to have a look at how it works

zas: pong

hey

samj1912 made a test, using paco for sir/solr and williams as db

i think we could get a huge speed up using a db slave on the same machine as sir/solr

and even further using unix socket instead of tcp to query the db

the whole indexing took something like 4 hours, but with significant network activity, and that's prolly the bottleneck, any thoughts ?

bitmap: ^^

hmm

I don't know how the indexing works tbh, I would expect it to query the data and build indexes concurrently

building the initial indexes that is

we could try it and see what effect it has

yes :)

samj1912: does sir need write access to the db ?

nope

for sir, it could query a slave, but the amqp triggers have to be on the master db

is this an issue ?

slave & master should be more or less in sync

I don't think so, there should be little to no lag

let's give it a try then, a dedicated slave db running on the same host, with unix socket access (through containers mounts)

and there will probably always be things in the queue, so the lag wouldn't actually matter

samj1912: step 1, move sir from paco to williams for tests

can we just try with queen?

if you feel it is safe, i'm ok with it

in fact, that's my idea, for production

queen is fast, has the db slave, and ssd everywhere

we were testing sir on the production db before, without issues, so I think it should be fine

another thing: we start to lack of space on serge, most diskspace is used by ftp, i think williams would be a much better host for it, no urge though

bitmap: how can I test the live indexing part