Grep without duplicates?











up vote
3
down vote

favorite












I have multiple revisions of a text file in separate files in the same folder.



How can I grep all files in that folder without listing any duplicate of lines with identical text?










share|improve this question




















  • 1




    What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
    – don_crissti
    1 hour ago








  • 1




    An example would clarify the question.
    – Kusalananda
    57 mins ago






  • 1




    @Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
    – don_crissti
    30 mins ago










  • @Don_crissti Desired output? Filename: Can but does not have to.
    – neverMind9
    21 mins ago










  • If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
    – don_crissti
    19 mins ago















up vote
3
down vote

favorite












I have multiple revisions of a text file in separate files in the same folder.



How can I grep all files in that folder without listing any duplicate of lines with identical text?










share|improve this question




















  • 1




    What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
    – don_crissti
    1 hour ago








  • 1




    An example would clarify the question.
    – Kusalananda
    57 mins ago






  • 1




    @Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
    – don_crissti
    30 mins ago










  • @Don_crissti Desired output? Filename: Can but does not have to.
    – neverMind9
    21 mins ago










  • If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
    – don_crissti
    19 mins ago













up vote
3
down vote

favorite









up vote
3
down vote

favorite











I have multiple revisions of a text file in separate files in the same folder.



How can I grep all files in that folder without listing any duplicate of lines with identical text?










share|improve this question















I have multiple revisions of a text file in separate files in the same folder.



How can I grep all files in that folder without listing any duplicate of lines with identical text?







text-processing command-line grep






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 41 mins ago









Jeff Schaller

36.4k952120




36.4k952120










asked 1 hour ago









neverMind9

36511




36511








  • 1




    What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
    – don_crissti
    1 hour ago








  • 1




    An example would clarify the question.
    – Kusalananda
    57 mins ago






  • 1




    @Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
    – don_crissti
    30 mins ago










  • @Don_crissti Desired output? Filename: Can but does not have to.
    – neverMind9
    21 mins ago










  • If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
    – don_crissti
    19 mins ago














  • 1




    What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
    – don_crissti
    1 hour ago








  • 1




    An example would clarify the question.
    – Kusalananda
    57 mins ago






  • 1




    @Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
    – don_crissti
    30 mins ago










  • @Don_crissti Desired output? Filename: Can but does not have to.
    – neverMind9
    21 mins ago










  • If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
    – don_crissti
    19 mins ago








1




1




What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago






What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago






1




1




An example would clarify the question.
– Kusalananda
57 mins ago




An example would clarify the question.
– Kusalananda
57 mins ago




1




1




@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago




@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago












@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago




@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago












If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago




If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago










4 Answers
4






active

oldest

votes

















up vote
1
down vote













Pipe the result into sort to filter duplicates.



grep exampletext | sort -u





share|improve this answer

















  • 2




    You are missing the files that you are going to grep.
    – Nasir Riley
    1 hour ago






  • 3




    and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
    – Jeff Schaller
    42 mins ago


















up vote
0
down vote













If what you need is to find which file(s) match some text, use:



$ grep -rl 'text to find' ./dir


If you need only the first match of each file:



$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done


which will not print the name of the files matching, but will be fast.



Or:



$ find ../* -type f -exec sh -c '
a=$(sed -n "/echo/{p;q}" "$1");
[ "$a" ] && printf "%sn" "$1 : $a"
' findsh {} ;


If you need the filename also (separated with :).






share|improve this answer




























    up vote
    0
    down vote













    I use:



    grep test files* | puniq


    puniq is: perl -ne '$seen{$_}++ or print;'



    It is similar to sort -u but it does not sort the file.






    share|improve this answer

















    • 1




      This won't work for obvious reasons...
      – don_crissti
      31 mins ago


















    up vote
    0
    down vote













    I would use something like:



    #!/bin/bash
    echo "What is the base name to reduce to unique lines? "
    read filename
    for i in *$filename*; do
    cat $i >> tempfile
    cat tempfile | sort -u -o tempfile
    done


    or



    cat *filename* | sort -u



    The placement of the *'s is based on how you name your itterations.





    • When done read tempfile.

    • Rename to whatever.



    Use with caution in a test directory with test files first





    • Tailor to your needs.






    share|improve this answer























      Your Answer








      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "106"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      convertImagesToLinks: false,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: null,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














       

      draft saved


      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f484086%2fgrep-without-duplicates%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      4 Answers
      4






      active

      oldest

      votes








      4 Answers
      4






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes








      up vote
      1
      down vote













      Pipe the result into sort to filter duplicates.



      grep exampletext | sort -u





      share|improve this answer

















      • 2




        You are missing the files that you are going to grep.
        – Nasir Riley
        1 hour ago






      • 3




        and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
        – Jeff Schaller
        42 mins ago















      up vote
      1
      down vote













      Pipe the result into sort to filter duplicates.



      grep exampletext | sort -u





      share|improve this answer

















      • 2




        You are missing the files that you are going to grep.
        – Nasir Riley
        1 hour ago






      • 3




        and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
        – Jeff Schaller
        42 mins ago













      up vote
      1
      down vote










      up vote
      1
      down vote









      Pipe the result into sort to filter duplicates.



      grep exampletext | sort -u





      share|improve this answer












      Pipe the result into sort to filter duplicates.



      grep exampletext | sort -u






      share|improve this answer












      share|improve this answer



      share|improve this answer










      answered 1 hour ago









      Jon Reinhold

      828614




      828614








      • 2




        You are missing the files that you are going to grep.
        – Nasir Riley
        1 hour ago






      • 3




        and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
        – Jeff Schaller
        42 mins ago














      • 2




        You are missing the files that you are going to grep.
        – Nasir Riley
        1 hour ago






      • 3




        and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
        – Jeff Schaller
        42 mins ago








      2




      2




      You are missing the files that you are going to grep.
      – Nasir Riley
      1 hour ago




      You are missing the files that you are going to grep.
      – Nasir Riley
      1 hour ago




      3




      3




      and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
      – Jeff Schaller
      42 mins ago




      and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of sort -u...
      – Jeff Schaller
      42 mins ago












      up vote
      0
      down vote













      If what you need is to find which file(s) match some text, use:



      $ grep -rl 'text to find' ./dir


      If you need only the first match of each file:



      $ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done


      which will not print the name of the files matching, but will be fast.



      Or:



      $ find ../* -type f -exec sh -c '
      a=$(sed -n "/echo/{p;q}" "$1");
      [ "$a" ] && printf "%sn" "$1 : $a"
      ' findsh {} ;


      If you need the filename also (separated with :).






      share|improve this answer

























        up vote
        0
        down vote













        If what you need is to find which file(s) match some text, use:



        $ grep -rl 'text to find' ./dir


        If you need only the first match of each file:



        $ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done


        which will not print the name of the files matching, but will be fast.



        Or:



        $ find ../* -type f -exec sh -c '
        a=$(sed -n "/echo/{p;q}" "$1");
        [ "$a" ] && printf "%sn" "$1 : $a"
        ' findsh {} ;


        If you need the filename also (separated with :).






        share|improve this answer























          up vote
          0
          down vote










          up vote
          0
          down vote









          If what you need is to find which file(s) match some text, use:



          $ grep -rl 'text to find' ./dir


          If you need only the first match of each file:



          $ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done


          which will not print the name of the files matching, but will be fast.



          Or:



          $ find ../* -type f -exec sh -c '
          a=$(sed -n "/echo/{p;q}" "$1");
          [ "$a" ] && printf "%sn" "$1 : $a"
          ' findsh {} ;


          If you need the filename also (separated with :).






          share|improve this answer












          If what you need is to find which file(s) match some text, use:



          $ grep -rl 'text to find' ./dir


          If you need only the first match of each file:



          $ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done


          which will not print the name of the files matching, but will be fast.



          Or:



          $ find ../* -type f -exec sh -c '
          a=$(sed -n "/echo/{p;q}" "$1");
          [ "$a" ] && printf "%sn" "$1 : $a"
          ' findsh {} ;


          If you need the filename also (separated with :).







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered 52 mins ago









          Isaac

          9,70611445




          9,70611445






















              up vote
              0
              down vote













              I use:



              grep test files* | puniq


              puniq is: perl -ne '$seen{$_}++ or print;'



              It is similar to sort -u but it does not sort the file.






              share|improve this answer

















              • 1




                This won't work for obvious reasons...
                – don_crissti
                31 mins ago















              up vote
              0
              down vote













              I use:



              grep test files* | puniq


              puniq is: perl -ne '$seen{$_}++ or print;'



              It is similar to sort -u but it does not sort the file.






              share|improve this answer

















              • 1




                This won't work for obvious reasons...
                – don_crissti
                31 mins ago













              up vote
              0
              down vote










              up vote
              0
              down vote









              I use:



              grep test files* | puniq


              puniq is: perl -ne '$seen{$_}++ or print;'



              It is similar to sort -u but it does not sort the file.






              share|improve this answer












              I use:



              grep test files* | puniq


              puniq is: perl -ne '$seen{$_}++ or print;'



              It is similar to sort -u but it does not sort the file.







              share|improve this answer












              share|improve this answer



              share|improve this answer










              answered 49 mins ago









              Ole Tange

              11.8k1448105




              11.8k1448105








              • 1




                This won't work for obvious reasons...
                – don_crissti
                31 mins ago














              • 1




                This won't work for obvious reasons...
                – don_crissti
                31 mins ago








              1




              1




              This won't work for obvious reasons...
              – don_crissti
              31 mins ago




              This won't work for obvious reasons...
              – don_crissti
              31 mins ago










              up vote
              0
              down vote













              I would use something like:



              #!/bin/bash
              echo "What is the base name to reduce to unique lines? "
              read filename
              for i in *$filename*; do
              cat $i >> tempfile
              cat tempfile | sort -u -o tempfile
              done


              or



              cat *filename* | sort -u



              The placement of the *'s is based on how you name your itterations.





              • When done read tempfile.

              • Rename to whatever.



              Use with caution in a test directory with test files first





              • Tailor to your needs.






              share|improve this answer



























                up vote
                0
                down vote













                I would use something like:



                #!/bin/bash
                echo "What is the base name to reduce to unique lines? "
                read filename
                for i in *$filename*; do
                cat $i >> tempfile
                cat tempfile | sort -u -o tempfile
                done


                or



                cat *filename* | sort -u



                The placement of the *'s is based on how you name your itterations.





                • When done read tempfile.

                • Rename to whatever.



                Use with caution in a test directory with test files first





                • Tailor to your needs.






                share|improve this answer

























                  up vote
                  0
                  down vote










                  up vote
                  0
                  down vote









                  I would use something like:



                  #!/bin/bash
                  echo "What is the base name to reduce to unique lines? "
                  read filename
                  for i in *$filename*; do
                  cat $i >> tempfile
                  cat tempfile | sort -u -o tempfile
                  done


                  or



                  cat *filename* | sort -u



                  The placement of the *'s is based on how you name your itterations.





                  • When done read tempfile.

                  • Rename to whatever.



                  Use with caution in a test directory with test files first





                  • Tailor to your needs.






                  share|improve this answer














                  I would use something like:



                  #!/bin/bash
                  echo "What is the base name to reduce to unique lines? "
                  read filename
                  for i in *$filename*; do
                  cat $i >> tempfile
                  cat tempfile | sort -u -o tempfile
                  done


                  or



                  cat *filename* | sort -u



                  The placement of the *'s is based on how you name your itterations.





                  • When done read tempfile.

                  • Rename to whatever.



                  Use with caution in a test directory with test files first





                  • Tailor to your needs.







                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited 39 mins ago

























                  answered 1 hour ago









                  Michael Prokopec

                  47112




                  47112






























                       

                      draft saved


                      draft discarded



















































                       


                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f484086%2fgrep-without-duplicates%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      Popular posts from this blog

                      Willebadessen

                      Ida-Boy-Ed-Garten

                      Residenzschloss Arolsen