#musicbrainz-devel

the rate limiter runs on a different IP

Ah, OK.

I wondered because the ratelimiter access is the only part where the search server directly deals with sockets.

The normal requests are handled by the framework.

yeah, it was a good consideration

is there a way from a java program to invoke the GC?

System.GC() apparently.

the openjdk management console also has a button for that, if you're using that

Or System.gc(), rather.

well, does anyone think that invoking it every 60 seconds is a bad idea?

Perhaps? :)

Mineo: using sun's jdk atm

ah

I’d like to know if running GC in a stuck state removes the CLOSE_WAIT sockets.

(Via destructors, or what they are called in Java.)

ijabz2: can you please do me a favor?

chirlu thats a good point because I forgot how that code works and lookong at it Im not closing the Datagram socket

can you create a new URL endpoint that runs the GC?

/run-gc-now-you-piece-of-shit

:)

then we can hit that URL whenever we want the GC to run

then we can see clearly what happens.

ijabz2: Yes, I saw that, but if it connects to a different IP address than what we see, it can’t be that.

ok, I;ll do that

thanks

What we see in the netstat output, I mean.

CPU usage is climing steadily.

roobarb is fine (restarted recently)

Gc may release memory used by lucene for the memory mapping of indexes but not sure if will as they are not on the heap

either way even if it does will just slow searches a little rather breaking anyhting so woth a try

it sure seems like the GC isn't doing its job

this may not help if the GC gets stuck.

but if the GC doesn't fire, it might

though running the GC on a regular basis may it getting stuck in the first place

Gentlecat: hm, the metabrainz sign up thing asks for an email address, couldn't it actually request that info from the mb account when doing oauth?

yeah, I can add that

I'd say auto-fill the form, but allow users to change it.

(I imagine quite a few people wanting to sign up won't actually be active editors, so they'll have to create an mb account, then sign up "again" (from their point of view))

ruaok ok ive done it, invoked using paramter gc=true

yeah, I was thinking that

on any request or a specifc endpoint, ijabz2?

Paul Taylor: Close datagram socket after contacting rate limiter

Like this search.musicbrainz.org/?gc=true

only from local machine, like we do frpo reloading indexes etc, is that ok

perfect

compiling now

\nick siddharth

s0014: Forward slash. :)

Yeah, by mistake. Sorry to spam the list.

Paul Taylor: Added gc option to force garbage collection

heh. ijabz2 my build succeeded. how should I proceed?

What do yo mean, same as usual I guess

your build failed.

oh, I see

erm, I cant see any errors details on Jenkins ?

ruaok I thought there was a way to force Jenkins but cant see it, instead done a tiny bit of refactoring and done a new checkin

doing a new pull

Hey, I am Siddharth Gupta, a final year BTech student. In GSOC'14, I did a Data Visualization project for CiviCRM. I was going through the Idea page, documentation as we as How to contribute. After reading some of the code, I am starting on setting up the server. I would like to know is there anything you are expecting us to do before we start talking about the project.

ijabz2: build succeeded. I'll deploy this.

Project search_server build #45: FIXED in 7 min 31 sec: http://ci.musicbrainz.org/job/search_server/45/

Paul Taylor: Tidy up of admin commands

invoking gc seems to have no effect on memory usage.

which is what I would roughly expect.

But we don’t have any CLOSE_WAIT sockets now, do we?

correct, we have none

I may need to wait a few hours for that to occur.

because a watched server never crashes

ruaok, classic bad timing Ive just had some feedback on the other bug (TreeMap Null) causing me to commit another change which will get us the neccessary debeugging if/when it happens again

Wil need another build and deploy, but Ill leave it you to decide if yo want to do it now or some other time

lets check one thing first...

what log file should I see your log messages in?

catalina.todaysdate.log

I'm seeing a lot of

catalina.2015--03-05.log i think

queryparser exceptions.

seems new

and there are a lot of them.

Right, we didnt use to log these we just returned the error to user

hmm, looks like an empty request to me

05-Mar-2015 16:04:38.311 WARNING [http-nio-8080-exec-16] org.musicbrainz.search.servlet.SearchServerServlet.doGet 9044705516.7897105009 AND artist:sri sai telicom maupakar/prakhar yamraaj:Cannot parse '9044705516.7897105009 AND artist:sri sai telicom maupakar/prakhar yamraaj': Lexical error at line 1, column 74. Encountered: <EOF> after : "/prakhar yamraaj"

the slash?

Project search_server build #46: SUCCESS in 8 min 8 sec: http://ci.musicbrainz.org/job/search_server/46/

Paul Taylor: Log query whatever error occurs instead of just if get a parseexception in case it is the query causing the error (which I think it is in this case of SEARCH-411)

Leo_Verto yes it doesnt like the slash does it

just a bad client then?

is that a http restriction not a lucene one

no clue

I think /.../ is a regex to Lucene.

ah yeah I was looking at lucene 3, thanks chirlu

maybe we don't want to log the queryparter errors.

the logs have become more spammy now.

not useful

Just got a timeout.

WFM

Happens only sometimes.

Anyway, the query demonstrates the /regex/ syntax. :)

ruaok I could log invalid queries that fail parseexception, but not log the stacktrace bit

just skip invalid queries entirely

I could also make that simpler configurable

Okay I can remove if you think not useful at all

yeah. not useful.

ruaok okay done

invalid queries might be a useful dev log, but if we're running arbitrary user-supplied queries... :)

thx

kepstin-laptop: Yes, in particular for advanced search with its many escape-needing special characters …

Paul Taylor: Do not log invalid queries that cannot be parsed.

ijabz2: deployed. first error: https://gist.github.com/mayhem/105e819cebe0e9fb...

ruaok can you get the preious line that shows the query like you did before please

ops sorry

05-Mar-2015 16:50:40.726 WARNING [http-nio-8080-exec-41] org.musicbrainz.search.servlet.SearchServerServlet.doGet artist:pandora /reyli barba recording:Solo el y yo/Alguien llena mi lugar:null

ruaok oops no sorry my mistake, its there

mhm, serverporn https://imgur.com/tTj1Y3P

ruaok it those slashes again , causing the treemap error as wel

ijabz2: is that useful info?

yeah

yeeeargh: give me all the ritter!

on both servers are in rotation. one has a new jar, the other one is on the auto killer.

when I get home I'm going to shove all the traffic down the one with the new jar in hopes of inducing the error and then hitting it with a gc command

okay, Im out for a few hours myself now

ruaok: http://www.ritter-sport.de/blog/wp-content/uplo... are you sure?

[17:28] <yeeeargh> ruaok: http://www.ritter-sport.de/blog/wp-content/uplo... are you sure?

FUCK TO THE NO.

<s0014> Hey, I am Siddharth Gupta, a final year BTech student. In GSOC'14, I did a Data Visualization project for CiviCRM. I was going through the Idea page, documentation as we as How to contribute. After reading some of the code, I am starting on setting up the server. I would like to know is there anything you are expecting us to do before we start talking about the project.

hi s0014 :)

Hi derwin, I have been looking at the issue tracker, and would now to understand what all I should do before discussing the ideas I have for the project.