#metabrainz

How do you build search indexes on a standalone server?

nelgin: we don't rely on one person for a whole project, but one person is mostly familiar with the docker part of it - I don't think that's weird in a team of 3 (for MB proper)

I'm not asking about docker, I'm asking about running the server standalone now.

Well, that's not just one person :) (although most people seem to think that's a lot more annoying than docker)

I assume you set up PostgreSQL authentication as indicated in INSTALL.md?

I feel like the instructions are half finished. Do I need rabbitmq for example? Anything special for searched?

I can do a direct database query

in the web interface and it works.

You need to set up search separately, but that shouldn't matter for InitDB

But then, if you have a database working, then you don't need to InitDB?

It did tell me "Creating search indexes" but it looked like it only took about 4 minutes.

Tue Apr 27 20:55:37 2021 : Creating EAA views ... (eaa/CreateViews.sql)

Tue Apr 27 20:55:37 2021 : Creating search indexes ... (CreateSearchIndexes.sql)

Tue Apr 27 20:59:03 2021 : Setting up replication ... (ReplicationSetup.sql)

Those are direct search indexes

Not SOLR

There's nothing in the README.md file that mentions SOLR.

INSTALL.md excuse me.

Sure, because this is the standalone musicbrainz server without a search server

For development, that's generally enough (I don't run a search server locally)

It's not for development.

I need the search server then.

You can install the search server locally too, yeah

mb_solr ?

mb-solr rather

I'm not sure that'll be less effort than waiting for yvanzo to check the docker stuff a bit more, but :)

https://github.com/metabrainz/mb-solr and https://github.com/metabrainz/sir

I need both? ok.

Those are linked from INSTALL.md, although admittedly only here:

Well, it'll be faster on my linux hardware than a vm on my windows machine.

"The server by itself doesn't rate limit any request it handles. If you're receiving 503s, then you're likely performing search queries without having set up a local instance of the search server along with the search index rebuilder. By default, search queries are sent to search.musicbrainz.org and are rate limited.

Once you set up your own instance, change SEARCH_SERVER in lib/DBDefs.pm to point to it."

Yeah, I saw that, but you'd think it would give me results then, but it didn't.

Should I do solr or sir first or doesn't it matter?

I forget whether you need to activate the default

# sub SEARCH_SERVER { "search.musicbrainz.org" }

# sub SEARCH_ENGINE { "SOLR" }

To get results at all tbh

Ah that would make sense.

I don't think it matters, re: order

https://sir.readthedocs.io/en/latest/setup/inde... indicates what you need re: rabbitmq too

THis all could have been done with php and a bit of javascript :)

And also indicates how and when you should need https://github.com/metabrainz/mbsssss which is the third chunk of the search server

Thanks for the help.

nelgin: we use docker in production too, but with a far higher index_limit: 2000000

Is the "Index limit exceeded" error accompanied with other log lines? It can be it hits this limit in very specific cases only we should address.

We just don't use docker-compose in production (musicbrainz-docker should be better named musicbrainz-docker-compose).

If you don’t use docker, you won’t have to set container’s resources at least. That can solve some issues when they were not set appropriately, that can also make the website less responsive during reindex.

And yes, it is certainly faster to use a Linux host (docker or not) than a VM on Windows. :)

The first hurdle is that there doesn't appear to be pip for phython2. Then I'm wondering, why is it still using python2?

no good reason, it is just late with conversion to python3, which happens to not be straightforward in this case.

Ok, I have pip for python2 so no biggie there.

ImportError: /usr/local/lib/python2.7/dist-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked

Seen that before?

python2 -m sir

no

I think I'll give up and rely on discogs instead.

ujson is part of requirements.txt, did you install them?

Collecting ujson==1.35

says ujson-1.35 was successfully installed.

It seems recent Ubuntu updates broke old ujson, this patch may solve it: https://github.com/ultrajson/ultrajson/issues/3...

Yeah, I was just reading that.

Thought it doesn't really tell you how to rebuild it. Python isn't my thing

I really really give up. I might try the docker on linux.

The docker image for sir uses an older version of ubuntu, that explains this error has never been seen before.

TIme to join the 21st centrury :)

I have a 12 core linux box with 32gb of memory. Can you suggest some sane parameters that'll give me some good performance? It's got 27gb free after flushing the cache so not much in use.

Err:1 http://security.debian.org/debian-security buster/updates InRelease

Temporary failure resolving 'security.debian.org'

Doesn't look good.

Copy default/indexer.ini and set index_limit to 2000000: https://github.com/metabrainz/musicbrainz-docke...

Any ideas on the network disconnect between the docker and the outside world?

I have ufw turned off so only using iptables

and not blocking outgoing connections anyway

Ah, this is Ubuntu's stupid DNS at work

"Docker manipulates iptables rules to provide network isolation. While this is an implementation detail and you should not modify the rules Docker inserts into your iptables policies" https://docs.docker.com/network/iptables/

if systemd-resolved is running it created this stupid 127.0.0.53 ip with the resolves behind it somehow. It's a mess. I'm about to fix it.

I have Linux distros that think they're smarter than me and think they know what is best.

Ok, for sane parameters: Bump db/search resources to 8GB/8g: https://github.com/metabrainz/musicbrainz-docke...

Nope, that dns thing didn't fix it either.

Can I add anything to docker-compose build to give more verbose output or logging?

No but you can probe the network from a temporary docker container: sudo docker run --rm metabrainz/python:2.7-20210115 bash

Well, it said there was no image so downloaded on and when I run the command again I just get my prompt back

Sorry: sudo docker run -it --rm metabrainz/python:2.7-20210115 bash

OK, there we go.

From the linux host system, you can also see the details of docker networks: sudo docker network ls; sudo docker network inspect bridge; sudo docker network inspect musicbrainz-docker_default

ping -c3 musicbrainz.org

from the python2-7 shell?

nothing.

can't ping by ip either.

There's my network output

Your bridge is missing gateway

I'll take your word for it...how do I fix it...or rather, how do you fix it so it doesn't break for someone else? :)

try restarting docker daemon first

Ok, just a sec.

sudo service docker restart # or docker.io

"Subnet": "172.17.0.0/16",

"Gateway": "172.17.0.1"

That looks better. Let me give it a ride.

Yes, much happier.

Add that to the faq :)

built. not bad. 12 minutes.

https://github.com/metabrainz/musicbrainz-docke... the "and finally" link goes to the wrong place. It leads to bind mounts.

actually --v or --mount - nothing about setting variables.

yvanzo you still there? I ran sudo docker-compose run --rm musicbrainz createdb.sh -fetch "ftp://64.50.236.52/pub/musicbrainz"

it downloaded the files then...

08006 DBI connect('dbname=musicbrainz_db;host=db;port=5432','musicbrainz',...) failed: FATAL: database "musicbrainz_db" does not exist

Ah fun, docker filled my /var/lib up. How the heck am I going to fix that.

df -h "$( sudo docker system info --format '{{.DockerRootDir}}' )"

Oh, i know for sure my /var filled up. I just created a new lvm for /var/lib/docker and mounted it so giving it another try.

Wed Apr 28 08:08:35 2021 : Creating database 'musicbrainz_db'

I just want to get as far as building the search index so I can leave it and go to bed. :)

[musicbrainz-docker] yvanzo opened pull request #191 (master…troubleshooting): Add troubleshooting page for frequent setup issues https://github.com/metabrainz/musicbrainz-docke...

why you should be using signal: https://signal.org/bigbrother/central-californi...

ok, db creation done...what's the next fun bit...

ruaok, can you take a look again at the consul changes PR you reviewed yesterday. beta is not starting with those changes and i am unable to figure out what's wrong.

oh, i think just figured it out.

what was it?

good thing that bug was fixed. previously that error would take down ALL sites. lol

it still doesn't work, looks like there is another bug somewhere. :-(

i started the master on beta and that works fine so other services are working properly.