9:11 AM
agentsim has quit
2017-06-05 15626, 2017
9:14 AM
ZaphodBeeblebrox
\o heisan gcilou
2017-06-05 15620, 2017
9:15 AM
ZaphodBeeblebrox is now known as CatQuest
2017-06-05 15613, 2017
9:16 AM
CatQuest
sorry that of all the scandinavian countries you had to go to sweden :D
2017-06-05 15627, 2017
9:16 AM
CatQuest looks out on the pretty sun here >:D
2017-06-05 15606, 2017
9:18 AM
ruaok
2017-06-05 15623, 2017
9:19 AM
gcilou
Lol the sun is finally peeking out
2017-06-05 15658, 2017
9:19 AM
gcilou
But man, these trains are difficult to figure out
2017-06-05 15604, 2017
9:21 AM
ruaok
the t-banen?
2017-06-05 15627, 2017
9:21 AM
ruaok
or trying to figure out why the swedish trains all look fugly?
2017-06-05 15628, 2017
9:21 AM
gcilou
All of them. Especially with my non-existent swedish
2017-06-05 15653, 2017
9:21 AM
CatQuest
I'm sure swedes speak english
2017-06-05 15613, 2017
9:22 AM
gcilou
Their signs don't
2017-06-05 15630, 2017
9:22 AM
ruaok
2017-06-05 15612, 2017
9:23 AM
ruaok
do you have a working mobile phone? google translate is a life saver.
2017-06-05 15630, 2017
9:23 AM
CatQuest
2017-06-05 15632, 2017
9:23 AM
ruaok
but you can stop any swede and ask. not only will they speak english, but they will be glad to help.
2017-06-05 15641, 2017
9:23 AM
ruaok
or take you to ikea, which may not be what you want.
2017-06-05 15606, 2017
9:24 AM
CatQuest
stay wy from ikea! :O
2017-06-05 15614, 2017
9:24 AM
CatQuest
(although the meatballs are jum)
2017-06-05 15614, 2017
9:24 AM
gcilou
True. The train and bus buying system is weird
2017-06-05 15616, 2017
9:24 AM
zag joined the channel
2017-06-05 15646, 2017
9:24 AM
ruaok
CatQuest: good think we don't operate a hadoop cluster
2017-06-05 15614, 2017
9:25 AM
CatQuest
well I read the word "elasticsearch" in there so I figured.. better
2017-06-05 15622, 2017
9:25 AM
CatQuest
share it just in case?
2017-06-05 15631, 2017
9:25 AM
CatQuest
/it might be insteresting anyway
2017-06-05 15633, 2017
9:26 AM
CatQuest
hmm gcilou I don't know if swedish systems are that different from norwegian ones.. whats weird about them? :D
2017-06-05 15634, 2017
9:29 AM
ruaok
they run on time? there are so many trains in a day?
2017-06-05 15629, 2017
9:30 AM
iliekcomputers
hmm, trains running on time, I wonder how that works
2017-06-05 15643, 2017
9:30 AM
iliekcomputers
:)
2017-06-05 15614, 2017
9:31 AM
zag
just for info, ever since musicbrainz-mirror went down last month. The main MB web service has also been unusable. I think a few big users switched back and its overloading again
2017-06-05 15617, 2017
9:31 AM
ruaok
it even works in spain. :)
2017-06-05 15627, 2017
9:31 AM
Guest29341 has quit
2017-06-05 15628, 2017
9:31 AM
ruaok
zag: we
2017-06-05 15637, 2017
9:31 AM
ruaok
're aware and working on it.
2017-06-05 15659, 2017
9:31 AM
zag
thx, any news about api keys or usage stats? I know that was a hope one day
2017-06-05 15606, 2017
9:32 AM
ruaok
not yet, no
2017-06-05 15629, 2017
9:32 AM
zag
k thx
2017-06-05 15619, 2017
9:36 AM
gcilou
Lol I guess just buying tickets for a ride rather than a certain number of days is different than in the US
2017-06-05 15609, 2017
9:38 AM
Matthew__ joined the channel
2017-06-05 15633, 2017
9:38 AM
Matthew__ is now known as Guest79527
2017-06-05 15630, 2017
9:40 AM
Guest79527
Hi, all. We have an issue where replication is permanently stuck in 'Continuing a previously aborted load' but no statements are executed and dbmirror_pending does not get cleared. This started happening at sometime between 1600 and 1700 BST yesterday. Do you have any advice as to how we might 'reset' replication? Can I safely truncate the dbmirror_pending table for example?
2017-06-05 15627, 2017
9:41 AM
Guest79527 is now known as MatthewGlubb
2017-06-05 15602, 2017
9:43 AM
reosarevok
bitmap, ruaok ^
2017-06-05 15652, 2017
9:43 AM
ruaok
hmmm, I don't know what might causes that.
2017-06-05 15610, 2017
9:44 AM
ruaok
but truncating the table might cause the replication to surely fail. which is may already have.
2017-06-05 15627, 2017
9:44 AM
ruaok
do you have logs going back that you could look at and see what happened?
2017-06-05 15641, 2017
9:44 AM
SothoTalKer
wow, sweden looks ugly '-'
2017-06-05 15611, 2017
9:45 AM
MatthewGlubb
It happened across all of our replication instances, production and pre-production databases - there was no user input. What I did see was that the previous packet did not complete applying all of its statements.
2017-06-05 15625, 2017
9:47 AM
bitmap
I saw someone filed MBS-9366 just now, which sounds the same. but I don't know what could've caused this yet
2017-06-05 15626, 2017
9:47 AM
BrainzBot
2017-06-05 15629, 2017
9:48 AM
MatthewGlubb
Thanks. That certainly looks like it could be the same problem as it matches the time the error started.
2017-06-05 15634, 2017
9:48 AM
MatthewGlubb
(for us)
2017-06-05 15637, 2017
9:48 AM
bitmap
especially in the artist_alias table
2017-06-05 15607, 2017
9:51 AM
bitmap
104949 would be the problematic packet
2017-06-05 15614, 2017
9:51 AM
MatthewGlubb
Yes
2017-06-05 15635, 2017
9:51 AM
MatthewGlubb
"artist_alias_idx_primary" UNIQUE, btree (artist, locale) WHERE primary_for_locale = true AND locale IS NOT NULL
2017-06-05 15632, 2017
9:53 AM
MatthewGlubb
So it seems like the artist/locale is being modified but it's trying to modify it to an existing artist/locale
2017-06-05 15610, 2017
9:54 AM
bitmap
2017-06-05 15639, 2017
9:54 AM
to81 joined the channel
2017-06-05 15648, 2017
9:54 AM
bitmap
that's in the dbmirror_pendingdata for that packet, the first and last sequences there are certain to conflict
2017-06-05 15601, 2017
9:55 AM
MatthewGlubb
Certainly looks that way
2017-06-05 15618, 2017
9:55 AM
bitmap
what's bizarre is that the artist IDs are different in the 'key' rows
2017-06-05 15613, 2017
9:56 AM
bitmap
never seen anything like that
2017-06-05 15654, 2017
9:56 AM
ruaok
fun. :(
2017-06-05 15614, 2017
9:57 AM
bitmap
yeah
2017-06-05 15614, 2017
9:58 AM
bitmap
2017-06-05 15600, 2017
9:59 AM
reosarevok
jesus2099, you broke everything!
2017-06-05 15610, 2017
9:59 AM
to81 has quit
2017-06-05 15629, 2017
9:59 AM
reosarevok
Got another supporter @ support asking about the same issue. Let me know as soon as we have a fix so I can tell them :)
2017-06-05 15641, 2017
9:59 AM
bitmap
but how the hell did dbmirror see both ids for the same sequence
2017-06-05 15611, 2017
10:01 AM
reosarevok tweets in the meantime
2017-06-05 15610, 2017
10:06 AM
D4RK-PH0ENiX has quit
2017-06-05 15644, 2017
10:06 AM
kyan has quit
2017-06-05 15654, 2017
10:06 AM
D4RK-PH0ENiX joined the channel
2017-06-05 15644, 2017
10:08 AM
agentsim joined the channel
2017-06-05 15647, 2017
10:11 AM
ZarkBit has quit
2017-06-05 15619, 2017
10:18 AM
bitmap
nvm, those artist IDs are different because they're update operations, sorry, I was thinking they were inserts
2017-06-05 15627, 2017
10:20 AM
bitmap
still alarming, but a lot less so :P
2017-06-05 15635, 2017
10:20 AM
drsaunders joined the channel
2017-06-05 15636, 2017
10:20 AM
bitmap
thanks coffee
2017-06-05 15616, 2017
10:22 AM
ruaok
lol
2017-06-05 15642, 2017
10:27 AM
zas
more IPs blocked, if someone complains, ask for IP first, we may have false positives on this batch
2017-06-05 15614, 2017
10:30 AM
SothoTalKer
wow, alomst 6000 hits to /collection/create
2017-06-05 15636, 2017
10:31 AM
SothoTalKer
and ~250k useragents from yahoo and bing
2017-06-05 15617, 2017
10:36 AM
to81 joined the channel
2017-06-05 15649, 2017
10:36 AM
ruaok
Gah. I knew crawlers would be a problem for us back in the day.
2017-06-05 15659, 2017
10:36 AM
ruaok
I didn't thin they'd still be a problem.
2017-06-05 15659, 2017
10:40 AM
to81 has quit
2017-06-05 15638, 2017
10:41 AM
to81 joined the channel
2017-06-05 15620, 2017
10:42 AM
agentsim has quit
2017-06-05 15657, 2017
10:42 AM
SothoTalKer
well, craw-delay is now set to 2 seconds for any bot except IA. that should make the hits per day to go to ~43200
2017-06-05 15621, 2017
10:43 AM
SothoTalKer
depending how many bots are run simultaneously
2017-06-05 15616, 2017
10:47 AM
ruaok
someone is working at hetzner today: > We have received your order and shall inform you once we have activated your request.
2017-06-05 15650, 2017
10:48 AM
SothoTalKer
yes, they are called "auto response bot" :D
2017-06-05 15608, 2017
10:51 AM
ruaok
one that takes 45 minutes to respond?
2017-06-05 15623, 2017
10:51 AM
reosarevok
German efficiency! :p
2017-06-05 15644, 2017
10:51 AM
ruaok
I can see the thinking.
2017-06-05 15604, 2017
10:52 AM
ruaok
we'll have one bot respond right away. and another that responds in 45 minutes.
2017-06-05 15612, 2017
10:52 AM
ruaok
yeah, people will think we work on holidays.
2017-06-05 15622, 2017
10:52 AM
SothoTalKer
45 mins is actually a very short response time.
2017-06-05 15632, 2017
10:52 AM
reosarevok
Devious!
2017-06-05 15643, 2017
10:52 AM
reosarevok
SothoTalKer: for a human yes, for a bot probably not :)
2017-06-05 15626, 2017
10:53 AM
SothoTalKer
their automatic response system is slightly overloaded because they got so many requests
2017-06-05 15635, 2017
10:53 AM
reosarevok
What is this, our ws?
2017-06-05 15637, 2017
10:53 AM
reosarevok
:p
2017-06-05 15622, 2017
10:54 AM
SothoTalKer
i bet it is just some poor person who has to work on holidays but can't actually do anything else than route your request to the right persons.
2017-06-05 15630, 2017
10:54 AM
ruaok
haters are gonna hate, sheesh.
2017-06-05 15655, 2017
10:59 AM
iliekcomputers
new servers?
2017-06-05 15647, 2017
11:00 AM
Rotab
moar!!
2017-06-05 15627, 2017
11:01 AM
SothoTalKer
i think you should use a server farm and not try to run websites like this from home! :X
2017-06-05 15639, 2017
11:01 AM
Freso
What if you live on a farm?
2017-06-05 15613, 2017
11:02 AM
SothoTalKer
animals don't count ^-^
2017-06-05 15607, 2017
11:08 AM
CatQuest
too many spam users @_@
2017-06-05 15625, 2017
11:08 AM
SothoTalKer
you could report them
2017-06-05 15643, 2017
11:08 AM
CatQuest
no, see yesterday's conversation
2017-06-05 15609, 2017
11:09 AM
CatQuest
hmph, why is crazy webcrawler still in this list :/
2017-06-05 15630, 2017
11:09 AM
SothoTalKer
because it takes up to 2 weeks until they read the new robots.txt
2017-06-05 15638, 2017
11:09 AM
CatQuest
hmmm
2017-06-05 15649, 2017
11:09 AM
SothoTalKer
same with all the other disallowed bots
2017-06-05 15653, 2017
11:09 AM
CatQuest
2017-06-05 15630, 2017
11:10 AM
SothoTalKer
indeed an admin could write them a mail and request an instant stop
2017-06-05 15656, 2017
11:10 AM
CatQuest
hmmm
2017-06-05 15657, 2017
11:10 AM
CatQuest
"Blocking our web crawler by IP address will not work. "
2017-06-05 15609, 2017
11:11 AM
ruaok
2017-06-05 15629, 2017
11:11 AM
ruaok
nice, that last set of blockings increased throughput by another ~1,000 requests/min
2017-06-05 15648, 2017
11:11 AM
SothoTalKer
if you write them a mail, they put your site on a blacklist the bot ignores.
2017-06-05 15606, 2017
11:12 AM
CatQuest
hey maybe we should?
2017-06-05 15648, 2017
11:12 AM
CatQuest
they tal kabout "if we crawl your siteto quickly"
2017-06-05 15623, 2017
11:13 AM
SothoTalKer
we blocked them totally via robots.txt so hits should gradually decline
2017-06-05 15643, 2017
11:13 AM
CatQuest
i already know that, i read the robots.txt
2017-06-05 15659, 2017
11:13 AM
ruaok
I wonder if we should only allow: Google, IA and then common crawl. if someone wants our data for their search engine, use common crawl
2017-06-05 15613, 2017
11:14 AM
SothoTalKer
if we still see the same amount of hits after 2 weeks, we could ask them nicely :)
2017-06-05 15625, 2017
11:14 AM
CatQuest
what is common crawl?
2017-06-05 15625, 2017
11:15 AM
CatQuest
also I womder what the "-" useragent/referrer is
2017-06-05 15631, 2017
11:15 AM
CatQuest
also top request "/"
2017-06-05 15649, 2017
11:15 AM
SothoTalKer
"/" is the main site
2017-06-05 15658, 2017
11:15 AM
CatQuest
basiclaly "no user agent/referrer" ?
2017-06-05 15604, 2017
11:16 AM
SothoTalKer
yes
2017-06-05 15610, 2017
11:16 AM
reosarevok
ruaok: I don't think we should block stuff like yandex completely (which seems has most of the market share in Russia)
2017-06-05 15619, 2017
11:16 AM
ruaok
2017-06-05 15621, 2017
11:16 AM
reosarevok
Slowing them should be good enough
2017-06-05 15624, 2017
11:16 AM
CatQuest
ah so / is internal links (someone going from some link inside mb to another page)
2017-06-05 15639, 2017
11:16 AM
SothoTalKer
some people might use referrer blockers or come from sites that link through those or come directly to MB
2017-06-05 15641, 2017
11:16 AM
CatQuest agrees with reo
2017-06-05 15649, 2017
11:16 AM
ruaok
reosarevok: I'm not denying them access to our data. I'm suggesting they get it via common crawl
2017-06-05 15601, 2017
11:17 AM
reosarevok
Sure, but is there any chance they're going to use that?
2017-06-05 15603, 2017
11:17 AM
Freso
CatQuest: "/" is someone going from the front page of MB to somewhere else on MB.
2017-06-05 15613, 2017
11:17 AM
ruaok
reosarevok: depends on how much they care to have our data.
2017-06-05 15617, 2017
11:17 AM
Freso
CatQuest: "-" is when there's no referrer info.
