#metabrainz

yvanzo[m]: I don't think it's that interesting to most people tbh. This post is mainly for people running mirrors right?

Yes, and editors, probably not worth mentioning it then.

[@zas](https://matrix.to/#/@zas666:matrix.org): Might be me but I did a curl HTTPS req to the new solr-priv host and it responded with a redirect to HTTP

Can you check, I'm a bit busy today trying to tie things up before my flight tomorrow

<02netizen1997> really dont know how to deal with “connection reset” error，can someone help me please？

got more context, netizen1997?

<02netizen1997> sorry for my poor english. i m using java with okhttpclient，here is my code

<02netizen1997> OkHttpClient client = new OkHttpClient().newBuilder() .callTimeout(10, TimeUnit.SECONDS) .connectTimeout(10, TimeUnit.SECONDS) .readTimeout(10, TimeUnit.SECONDS) .connectionPool(new ConnectionPool(10,10,TimeUnit.SECONDS)) .build(); String url = "https://musicbrainz.org/ws/2/release?query=lanterns"; // url =

"https://blog.csdn.net/super_kingking/article/details/70992012"; Request request = new Request.Builder() .url(url) .method("GET", null) .addHeader("User-Agent","album manager/1.0(xiangyu97@live.cn), personal use") .addHeader("Accept",

"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7") .addHeader("Accept-Encoding", "gzip, deflate, br, zstd") .addHeader("Accept-Language","zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6") .addHeader("Accept-Language","zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6") .build();

System.out.println(request.toString()); try { Response response = client.newCall(request).execute();

<02netizen1997> get

<02netizen1997> java.lang.RuntimeException: java.net.SocketException: Connection reset

not much of a java guy, but do you need all those options passed in the headers?

<02netizen1997> i m also a new beginner of java, i just dont know how to do, so i append too many options in my headers..

I'd say skip everything by the User-Agent header.

<02netizen1997> i have tried but faild.

anyone here with more java experience who could help?

<11techmorningstar> @netizen1997 sounds like a network issue to me.

<02netizen1997> but when i use python, it works without any config set...:lookdown:

<02netizen1997> java is too complex, i hate it, but i really need it.:lookdown:

<02netizen1997> It's ok to use my code sending a request to other website..

also netizen: please use a paste bin for that'

<02netizen1997> `

<02netizen1997> paste bin? means delete my message?

netizen1997: tried it locally and it works for me so likely a network issue.

rimskii: can you share the psql command you used on wolf?

<02netizen1997> thank you very much, maybe i need a proxy? i will try it later

lucifer: the TfidfVectorizer refuses to run in parallel in threads. I think there is something blocking inside scikit learn.

mayhem[m]: code snippet?

I have a thread version of the same code now, should I check that in ?

<02netizen1997> shit!!! it works when i set proxy..

mayhem[m]: sure

<02netizen1997> appreciate again，now i know how to do my next work。

pushed

<02netizen1997> It' s ok when i use edge visit the api without setting proxy, so I always think It would also work when I use my code to send a request without proxy( in fact, it should workd, right?), there is no different when using browser or use code, right? So, it

mayhem: do you have the logs of this when you ran it?

really weird results.

mayhem: can you try this? https://github.com/amCap1712/fast-fuzzy

using threadpoolexecutor

oh why so?

I've never worked out how to make dynamic jobs work. let me read your code.

ah, I see how you do it.

<lucifer[m]> "rimskii: can you share the..." <- A minute

your code is running lucifer. 106% CPU/

how the threads are being started is not the problem (nor did I suspect that to be the problem)

hmm i see

the vectorizer function is where it all goes bad. but I've skimmed the source for it and I can't see why it would be a problem.

Can you change threalpoolexecutor to processpoolexecutor

And see if that changes anything,m

s/,m/?/

that does.

still highly variable times per chunk.

<lucifer[m]> "rimskii: can you share the..." <- `psql postgresql://musicbrainz:musicbrainz@localhost:5433/musicbrainz_db`

it using multiple cores though right?

yes

see wolf.

73% of CPU used

when I run

> \dt

mayhem[m]: omg mybe its because of me

rimskii: can you run `\dn`?

rimskii: no, that is me. :)

mayhem: i have a hunch that sklearn has internal locking or something to limit the number of cores the library is allowed to use

there is a library called joblib which can be used to change that

that is something that scientists would do. rather than learn computer science, lol.

oh wow i see what's going on

rimskii: if you are running it on wolf directly, use 5432 as the port

apparently there is another db running on wolf at 5433 port.

i'll see if i can find what is running that and shut it down

oh okay

done

thank you!

try again.

its working now!

thanks

5432 working

5433 is shut down

cool. to be clear, 5433 when you run locally, 5432 when you run on wolf.

lucifer[m]: that was for the MB area GSoC project IIRC, so not needed

makes sense

mayhem: spark has tf-idf vectorizer too btw

good to know.

the problem with your threadpool approach is that it saves the results in ram until everything is processed. which makes the memory footprint much worse.

can you try adding a del future inside the loop that processes the finished futures

added. let me run it over the whole data and see what happens. hard to tell when it might finish though.

or futures.remove(future) maybe

mayhem[m]: sounds good

[musicbrainz-server] 14mwiencek merged pull request #3294 (03beta…eaa-type-pot): Add `event_art_archive.art_type` to extract_pot_db https://github.com/metabrainz/musicbrainz-serve...

ha. your code was missing the thread_data = [] assignment, so that thread_data kept growing.

not having the growing thread data causes.... only one process to execute at a time.

oh lol 😅

ok, time to take a closer look at job control.

mayhem[m]: huh that is weird

there is deffo something weird.

I dont think the job control stuff is going to work. each of the indexes we build is tiny. so it probably never decides to use more than one thread.

mayhem: it is using 8 cores for me, running my script with the thread_data fix

check wolf

ok, interesting. see if it finishes in reasonable amount of time.

I need to get moving for today anways

s/anways/anyways/

free doesn't seem to point any major uptick in ram usage so far

yeah, it might not be as big as a problem as I suspect.

but for this POC, fine.

<06salaxceitor> hi! how are you guys doing

<06salaxceitor> came back after some time off and I trying to set up the development environment

<06salaxceitor> i am getting a blank screen with this errors in the browser

<06salaxceitor> - Failed to load resource: the server responded with a status of 404 (NOT FOUND)Understand this error indexPage.js:1Failed to load resource: the server responded with a status of 404 (NOT FOUND)Understand this error close_hover.png:1 Failed to load resource: net::ERR_CONNECTION_REFUSED

<06salaxceitor> no errors in the docker logs

Hello salaxceitor, which project are you trying to set up?

[musicbrainz-server] 14mwiencek opened pull request #3296 (03master…mbs-13615): MBS-13615: Skip same-entity relationships when batch-adding https://github.com/metabrainz/musicbrainz-serve...

atj: we get a 302 redirects on both https://solrcloud-sir & solrcloud-privileged, that's solr answering this 302

it returns http because the internal connection is done over http

mayhem: the statement that logged timings for each batch is wrong i think, due to which it appears that some indexes are built vastly slower than others

atj: I think we need something as described in this thread: https://www.eclipse.org/lists/jetty-users/msg10...

Has it finished building the whole index ever?

don't divide by recording_data

mayhem: no, it failed both times with ProcessPoolExecutor because nmslib indices cannot be pickled apparently. I can work on a version that save the indices to disk and load them in the main thread later.

Ahhh, pickling. I was wondering how it was going to do the process boundary transfer

I wonder if the vectorizer code is separable from the rest of scikit learn so that we can run without silly constraints

I guess this is what Boeing pilots must contend with when they fly Airbus....

lolol

But I suspect Airbus let their pilots fly on all engines if they want...

ran it on a subset of the data

and i think it is doing the processing in parallel.

mayhem: yes, it is doing everything in parallel. pressing H after opening top i see multiple theads.

Threads not procs? How did you accomplish that?

in top?

No, in the script

ah okay that, threadpoolexecutor

In top threads are shown as procs