loops - bash looping and extracting of the fragment of txt file

Question

I am dealing with the analysis of big number of dlg text files located within the workdir. Each file has a table (usually located in different positions of the log) in the following format:

File 1:

    CLUSTERING HISTOGRAM
    ____________________


________________________________________________________________________________
     |           |     |           |     |
Clus | Lowest    | Run | Mean      | Num | Histogram
-ter | Binding   |     | Binding   | in  |
Rank | Energy    |     | Energy    | Clus|    5    10   15   20   25   30   35
_____|___________|_____|___________|_____|____:____|____:____|____:____|____:___
   1 |     -5.78 |  11 |     -5.78 |   1 |#
   2 |     -5.53 |  13 |     -5.53 |   1 |#
   3 |     -5.47 |  17 |     -5.44 |   2 |##
   4 |     -5.43 |  20 |     -5.43 |   1 |#
   5 |     -5.26 |  19 |     -5.26 |   1 |#
   6 |     -5.24 |   3 |     -5.24 |   1 |#
   7 |     -5.19 |   4 |     -5.19 |   1 |#
   8 |     -5.14 |  16 |     -5.14 |   1 |#
   9 |     -5.11 |   9 |     -5.11 |   1 |#
  10 |     -5.07 |   1 |     -5.07 |   1 |#
  11 |     -5.05 |  14 |     -5.05 |   1 |#
  12 |     -4.99 |  12 |     -4.99 |   1 |#
  13 |     -4.95 |   8 |     -4.95 |   1 |#
  14 |     -4.93 |   2 |     -4.93 |   1 |#
  15 |     -4.90 |  10 |     -4.90 |   1 |#
  16 |     -4.83 |  15 |     -4.83 |   1 |#
  17 |     -4.82 |   6 |     -4.82 |   1 |#
  18 |     -4.43 |   5 |     -4.43 |   1 |#
  19 |     -4.26 |   7 |     -4.26 |   1 |#
_____|___________|_____|___________|_____|______________________________________

The aim is to loop over all the dlg files and take the single line from the table corresponding to wider cluster (with bigger number of slashes in Histogram column). In the above example from the table this is the third line.

   3 |     -5.47 |  17 |     -5.44 |   2 |##

Then I need to add this line to the final_log.txt together with the name of the log file (that should be specified before the line). So in the end I should have something in following format (for 3 different log files):

"Name of the file 1": 3 |     -5.47 |  17 |     -5.44 |   2 |##
"Name_of_the_file_2": 1 |     -5.99 |  13 |     -5.98 |  16 |################
"Name_of_the_file_3": 2 |     -4.78 |  19 |     -4.44 |   3 |###

A possible model of my BASH workflow would be:

#!/bin/bash
do
  file_name2=$(basename "$f")
  file_name="${file_name2/.dlg}"
  echo "Processing of $f..."
  # take a name of the file and save it in the log
  echo "$file_name" >> $PWD/final_results.log
  # search of the beginning of the table inside of each file and save it after its name
  cat $f |grep 'CLUSTERING HISTOGRAM' >> $PWD/final_results.log
  # check whether it works
  gedit $PWD/final_results.log
done

Here I need to substitute combination of echo and grep in order to take selected parts of the table.

Answer 1

You can use this one, expected to be fast enough. Extra lines in your files, besides the tables, are not expected to be a problem.

grep "#$" *.dlg | sort -rk11 | awk '!seen[$1]++'

grep fetches all the histogram lines which are then sorted in reverse order by last field, that means lines with most # on the top, and finally awk removes the duplicates. Note that when grep is parsing more than one file, it has -H by default to print the filenames at the beginning of the line, so if you test it for one file, use grep -H.

Result should be like this:

file1.dlg:   3 |     -5.47 |  17 |     -5.44 |   2 |##########
file2.dlg:   3 |     -5.47 |  17 |     -5.44 |   2 |####
file3.dlg:   3 |     -5.47 |  17 |     -5.44 |   2 |#######

---

Here is a modification to get the first appearence in case of many equal max lines in a file:

grep "#$" *.dlg | sort -k11 | tac | awk '!seen[$1]++'

We replaced the reversed parameter in sort, with the 'tac' command which is reversing the file stream, so now for any equal lines, initial order is preserved.

---

Second solution

Here using only awk:

awk -F"|" '/#$/ && $NF > max[FILENAME] {max[FILENAME]=$NF; row[FILENAME]=$0}
           END {for (i in row) print i ":" row[i]}' *.dlg

---

Update: if you execute it from different directory and want to keep only the basename of every file, to remove the path prefix:

awk -F"|" '/#$/ && $NF > max[FILENAME] {max[FILENAME]=$NF; row[FILENAME]=$0}
           END {for (i in row) {sub(".*/","",i); print i ":" row[i]}}'