<- Back
Comments (21)
- theamkI think the the second part was kinda obvious? The moment I read this:> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.it was pretty obvious I was not anything like him. My intuitive answers are pretty different.- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specific operation. For example, if there job starts by fetching data, and this fetch can be super slow, use 'Prefer Old' and put a timeout on the fetch part. This way your job won't be interrupted if the fetch just finally succeed minutes before next scheduled interval hit.Oh, and re "If the Prefer Old semantics are not offered, you can’t really emulate them using the two primitives of regular scheduling and limiting concurrency." - you totally can. Set concurrency to 2, and add as a first thing: "fetch the list of jobs running; if there is anyone except me, exit right away".Really, not that tricky at all.
- nosefrogMajor lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
- teleforceThe most popular resource manager for job submission and queueing system is Slurm. It's being used in majority of TOP500 supercomputers, and overwhelming majority of the world HPCs [1].SchedMD the leading developer of Slurm has recently being acquired by Nvidia, while Slurm remaining free and open source, but somehow it's Wikipedia entry is not yet updated accordingly.[1] Slurm Workload Manager:https://en.wikipedia.org/wiki/Slurm_Workload_Manager[2] Nvidia Acquires SchedMD (7 comments):https://news.ycombinator.com/item?id=46277190
- wewewedxfgdfYou solve this simply with two cron jobs, one for weekend and one weekdays.
- zdc1Tangential, but when dealing with queues, the first thing you want to do is have a basic grounding of queuing theory, and know whether you're optimising for throughput or worker utilisation (i.e. what are your SLAs and efficiency targets?). IME each goal involves fairly different metrics and scaling rules, so you'll want to know what you're prioritising.
- irjustinI remember learning about CSV parsing and how it's conceptually simple, yet beyond the simple , and quotes: the corner cases bloat your parser 10-15x.
- ktimespiI find it very annoying when queue problems break into queue-of-queue patterns like in the `wait`scenario
- zmjI haven't modeled it, but I wonder how far you'd get on randomizing the policy choice for concurrency limit 1. Maybe weighted by past results, but bounded to allow it to shift instead of falling permanently into a basin.
- Unified-Mentor[flagged]
- Unified-Mentor[flagged]