Savannah W-ITS

When you write perfect shell scripts you expect them to work, yes? 

I was recently working on a  project that used both the venerable KSH and relatively large loop (processing multiple folders containing several hundred files, with ~10 functions, with ~sequential, consecutive uses of sed -e ’s/old pattern/new pattern/g’, a smattering of calls to awk, perhaps some largish variable assignments (values of ~512kb char), etc.)   Things were great with my small test set (i.e. less than 10 files.)

Error/problem symptoms with larger file set:

• errors showed up at random points in the script (I tried to remove last change and the problem would still show up…)
• it was a hard exit (the script crashed)
• sometimes it looked like awk was the problem
• sometimes it looked like sed was the problem
• adjusting the script at or near the reported error ‘location’ might shift the error to a new location
• sometimes the indicated problem area was using:  $(call system tools),
  • i.e. NEW_VAR=”$(echo “${MYVAR}” | sed -e ’s/old_string/new__string/g’)”
• and sometimes it occurred during a script function call
• note that as I worked with the larger file set I needed more sed and/or awk statements so they were added until things started breaking…
  
• in call cases a string was being parsed and assigned to a variable
• I added debugging code showing current & previous files, last function call, etc.   After each run the file that it ‘crashed on’ would/might shift (in addition to pointing at the use of sed or awk…)

At this point I did an Internet search for:

ksh “job list” infinite loop

and found the link below.  Searching for the actual error message (below) did not yield quick results (but this post should now start showing up…)

KSH Problem Summary – a random hard crash of a KSH script when doing string manipulation while processing more than a few files in a loop.  The actual error message:

warning: job list infinite loop -- this should not happen

A very helpful message, yes? It appears that this was previously a problem and some recent (FC 11?) kernel changes brought it to light.  [[ You KNEW that those mystery scripting problems (where you switched shells/tools to resolve) were caused by Murphy, right? ]] 

Solution – update your version of KSH (see below.)

Snippet below from:  https://bugzilla.redhat.com/show_bug.cgi?id=518942

Comment #5 From Fedora Update System 2009-08-27 18:51:39 EDT ——-

ksh-20090630-1.fc11 has been pushed to the Fedora 11 testing repository.  If
problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update ksh'.  You can provide feedback
for this update here:
http://admin.fedoraproject.org/updates/F11/FEDORA-2009-9050

A possible work-around (perhaps you are stuck with the KSH version for some reason) is to use smaller sets of PIPEs…  While tinkering I noted that the problem started after adding a new sed statement to the pipe stream.  Thinking that it was a sed syntax problem just led to more head scratching. I have actually seen similar problems on systems with process utilization management enabled – your streaming task uses up N+processes and fails when your allocation of resources is consumed.  In that scenario the solution was to break the pipes down into smaller ‘pipe sets’, i.e.:

• VAR=$(cat < ${FILE} | pipe# 1 | pipe #2 | pipe #3 | etc)   and then
• …
• VAR_N=$(echo “${PREVIOUS_VAR}” | | pipe# 1 | pipe #2 | pipe #3)  and repeat as needed

You could also out the first stream to a new, temporary file, then start the next stream by reading that file while output to a third file, etc – lots of cleanup…

Related posts:

1. VMware under Fedora 8 Since the VMware install under Fedora 9 was problematic and...
2. Wolfram Alpha – sort of live… I happened to notice that the Wolfram Alpha team was...
3. Fedora 10 install problems Could just be me… Could just be luck of the...
4. Fedora 9 Yum error: no id element found Things are great when they work! Some recent change (around...
5. VMware under Fedora 9 (not) It should be noted that VMware Server 2.0 does NOT...