I would mark those not rechable first and only delete them in the second run
2006-05-09 12924, 2006
nikki_
yeah, same
2006-05-09 12938, 2006
ruaok
nikki_: simpl, base it on time.
2006-05-09 12952, 2006
nikki_
so you'd need a lastchecked thingy somewhere
2006-05-09 12959, 2006
ruaok
nope.
2006-05-09 12904, 2006
nikki_
hmm?
2006-05-09 12917, 2006
ruaok
time() mod (60 * 60 * 24)
2006-05-09 12932, 2006
ruaok
gives you day you're in.
2006-05-09 12944, 2006
nikki_ blinks
2006-05-09 12944, 2006
ruaok
mod that by 30 and you get a date 0 - 29
2006-05-09 12906, 2006
ruaok
on the first day, do the first 30th of the URLs
2006-05-09 12915, 2006
ruaok
on the second day, do the second 30th of the URLs
2006-05-09 12930, 2006
ruaok
your script, once invoked per day, would do the right think.
2006-05-09 12937, 2006
ruaok
some border conditions could be lost. eh.
2006-05-09 12942, 2006
nikki_ will take your word for it
2006-05-09 12946, 2006
ruaok
you'd get the next month.
2006-05-09 12912, 2006
ruaok hates keeping state of ANYTHING
2006-05-09 12943, 2006
nikki_
why?
2006-05-09 12948, 2006
Shepard
oh and of course wp, discogs, ... urls can be excluded
2006-05-09 12952, 2006
ruaok
its a pain in the arse.
2006-05-09 12954, 2006
yllona is listening to Frankie And Johnny by Brook Benton from Best of Brook Benton [Mercury] (1961) (0:13 / 2:28)
2006-05-09 12909, 2006
ruaok
just think of you having to rmeember things.
2006-05-09 12917, 2006
ruaok
how oftern do you forget?
2006-05-09 12920, 2006
wolfsong13 is listening to Leela James - Soul Food [from "A Change Is Gonna Come" (2005)]
2006-05-09 12932, 2006
ruaok
keeping track of state is always a pain
2006-05-09 12902, 2006
ruaok
that's why HTTP is a stateless protocol -- its easy and simple.
2006-05-09 12904, 2006
yllona switches up for wolf
2006-05-09 12908, 2006
yllona is listening to A Change Is Gonna Come by Sam Cooke from Ain't That Good News (1964) (0:06 / 3:14)
2006-05-09 12919, 2006
ruaok
keeping track of state in HTTP is cookies and all the compllicated mess.
2006-05-09 12940, 2006
Shepard thinks that way we would drop thousands of wikipedia urls just because there server is down when the script is running ;)
2006-05-09 12946, 2006
Shepard
*their
2006-05-09 12947, 2006
wolfsong13 wanders off to eat
2006-05-09 12948, 2006
ruaok
relax, we're not just going to delete them.
2006-05-09 12921, 2006
Shepard
but also bombing their server? ;)
2006-05-09 12929, 2006
Nyght
yeah, that's sorta why I personally think it's best to have a person checking them
2006-05-09 12938, 2006
Nyght
one was a really slow load, but it was present.. things like that.
2006-05-09 12909, 2006
ruaok
Shepard: > wait a second, do next one.
2006-05-09 12916, 2006
ruaok
common web practice.
2006-05-09 12923, 2006
ruaok
one hit per second is not bombing
2006-05-09 12927, 2006
Shepard thinks it doesn't work without a bit field keeping state
2006-05-09 12939, 2006
Shepard
no you misunderstand
2006-05-09 12944, 2006
ruaok
state of a url?
2006-05-09 12946, 2006
Shepard
the bombing was just a joke
2006-05-09 12952, 2006
Shepard
yes
2006-05-09 12905, 2006
Nyght
and I honestly have no idea how the program is "hitting" but it seems like it's about every fifteen seconds, or so.. I hope it's not making a load on the server.
2006-05-09 12912, 2006
ruaok
write a report: todays URLs that didn't answer [ Delete ]
2006-05-09 12920, 2006
Shepard
what do you want to do when the script checks a url and the server is not reachable?
2006-05-09 12927, 2006
Shepard
hm
2006-05-09 12953, 2006
Shepard
well I think a script could delete them automatically
2006-05-09 12918, 2006
ruaok
it could.
2006-05-09 12928, 2006
ruaok
and the script could be written to keep some state.
2006-05-09 12933, 2006
Nyght
I would imagine you would want to spot check the list at least.
2006-05-09 12941, 2006
ruaok
if the state happens to get lost, its start from the beginning.
2006-05-09 12958, 2006
Shepard
why should it get lost?
2006-05-09 12904, 2006
Shepard
do my albums in the db get lost, too?
2006-05-09 12904, 2006
ruaok
the feewer state files we need to keep track of, the fewer things we can screw up.
2006-05-09 12908, 2006
ruaok
thus is something recovers gracefully, we don't have to really care.
2006-05-09 12930, 2006
ruaok
thus leaving more time for the things that we do need to care about.
2006-05-09 12955, 2006
Shepard
if (however that should happen) the state gets lost, that's not critical
2006-05-09 12946, 2006
ruaok
ding
2006-05-09 12903, 2006
ruaok can live with that
2006-05-09 12911, 2006
ruaok
nikki_: are you done yet? :-)
2006-05-09 12923, 2006
ruaok
all this keeps distrating me from 1090
2006-05-09 12934, 2006
Shepard
pasta needs its time...
2006-05-09 12948, 2006
ruaok
as do regexps
2006-05-09 12918, 2006
yllona
shep: what type of pasta?
2006-05-09 12930, 2006
Shepard
I don't know, ask nikki
2006-05-09 12945, 2006
ruaok wasn't asking about the pasta
2006-05-09 12953, 2006
Shepard
I know
2006-05-09 12907, 2006
ruaok reminisces of the last pasta he ate. on a glacier a week ago
2006-05-09 12932, 2006
nikki_ hat keinen hunger mehr!
2006-05-09 12937, 2006
ruaok
yay
2006-05-09 12937, 2006
nikki_
yllona: farfalle
2006-05-09 12939, 2006
nikki_
well, it was
2006-05-09 12951, 2006
nikki_
now it's pasta-mush in my stomach :D
2006-05-09 12900, 2006
Shepard
du meinst, du bist satt?
2006-05-09 12905, 2006
nikki_
ja!
2006-05-09 12909, 2006
yllona
farfalle with waht type of sauce?
2006-05-09 12912, 2006
nikki_
tomato
2006-05-09 12919, 2006
Shepard
keinen hunger mehr haben would mean you don't want to eat it :)
2006-05-09 12945, 2006
yllona
nikki: tomato and ?
2006-05-09 12959, 2006
yllona apologizes for the specificity ;)
2006-05-09 12903, 2006
nikki_ wouldn't want to eat regurgitated pasta...
2006-05-09 12912, 2006
nikki_
er
2006-05-09 12930, 2006
nikki_
sun-dried tomato and herby stuff and whatever else!
but these herbs grow wild and/or natuarally in any "southern clime"
2006-05-09 12907, 2006
yllona
most of these herbs are everywhere in cali.. same for spain or italy. but the packaged dried herbs are really convenient. super tasty and work well for veggie, meat, pulry or fish.
2006-05-09 12920, 2006
yllona
*poultry or fish.
2006-05-09 12947, 2006
yllona
so yes, shep an essential for the ktichen :)
2006-05-09 12914, 2006
Shepard
yup
2006-05-09 12944, 2006
yllona
good choice
2006-05-09 12920, 2006
yllona remembers when shep was shyly asking questions about cooking
2006-05-09 12936, 2006
Shepard
I did?
2006-05-09 12944, 2006
yllona is amazed how quickly shep has learned
2006-05-09 12908, 2006
Shepard
oh no, those we have used here for years ;) I'm still not a big cook
2006-05-09 12908, 2006
yllona
shep: yes, ssenrupus was cooking a pie... it was a while gao
2006-05-09 12917, 2006
Shepard
oh yeah, I recall
2006-05-09 12907, 2006
yllona wants everyone to cook (well) it's so relaxing (and healthy). a major creative out let
2006-05-09 12931, 2006
nikki_
it's not relaxing or healthy when you're like me :/
2006-05-09 12937, 2006
nikki_
I made very bad cake!
2006-05-09 12949, 2006
nikki_
the first was ok, but it got worse
2006-05-09 12905, 2006
yllona
nikki_: you're a special case :)
2006-05-09 12913, 2006
nikki_
I guess so!
2006-05-09 12922, 2006
nikki_
I can boil water and make pasta!
2006-05-09 12937, 2006
nikki_
but pretty much everything I try and *cook* is a disaster
2006-05-09 12944, 2006
yllona
shep: all i'm gonna say on this matter -- all of the men i know are excellent cooks. mostly began for purely economic --- eating out is too costly
2006-05-09 12958, 2006
nikki_
apparently my supposed father was a good cook
2006-05-09 12908, 2006
nikki_
I didn't inherit his skills, it seems
2006-05-09 12926, 2006
Shepard
or it is cheap at the uni canteen but doesn't taste at all
