Grep without duplicates?
up vote
3
down vote
favorite
I have multiple revisions of a text file in separate files in the same folder.
How can I grep all files in that folder without listing any duplicate of lines with identical text?
text-processing command-line grep
add a comment |
up vote
3
down vote
favorite
I have multiple revisions of a text file in separate files in the same folder.
How can I grep all files in that folder without listing any duplicate of lines with identical text?
text-processing command-line grep
1
What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago
1
An example would clarify the question.
– Kusalananda
57 mins ago
1
@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago
@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago
If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago
add a comment |
up vote
3
down vote
favorite
up vote
3
down vote
favorite
I have multiple revisions of a text file in separate files in the same folder.
How can I grep all files in that folder without listing any duplicate of lines with identical text?
text-processing command-line grep
I have multiple revisions of a text file in separate files in the same folder.
How can I grep all files in that folder without listing any duplicate of lines with identical text?
text-processing command-line grep
text-processing command-line grep
edited 41 mins ago
Jeff Schaller
36.4k952120
36.4k952120
asked 1 hour ago
neverMind9
36511
36511
1
What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago
1
An example would clarify the question.
– Kusalananda
57 mins ago
1
@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago
@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago
If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago
add a comment |
1
What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago
1
An example would clarify the question.
– Kusalananda
57 mins ago
1
@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago
@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago
If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago
1
1
What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago
What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago
1
1
An example would clarify the question.
– Kusalananda
57 mins ago
An example would clarify the question.
– Kusalananda
57 mins ago
1
1
@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago
@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago
@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago
@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago
If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago
If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago
add a comment |
4 Answers
4
active
oldest
votes
up vote
1
down vote
Pipe the result into sort to filter duplicates.
grep exampletext | sort -u
2
You are missing the files that you are going togrep.
– Nasir Riley
1 hour ago
3
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose ofsort -u...
– Jeff Schaller
42 mins ago
add a comment |
up vote
0
down vote
If what you need is to find which file(s) match some text, use:
$ grep -rl 'text to find' ./dir
If you need only the first match of each file:
$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done
which will not print the name of the files matching, but will be fast.
Or:
$ find ../* -type f -exec sh -c '
a=$(sed -n "/echo/{p;q}" "$1");
[ "$a" ] && printf "%sn" "$1 : $a"
' findsh {} ;
If you need the filename also (separated with :).
add a comment |
up vote
0
down vote
I use:
grep test files* | puniq
puniq is: perl -ne '$seen{$_}++ or print;'
It is similar to sort -u but it does not sort the file.
1
This won't work for obvious reasons...
– don_crissti
31 mins ago
add a comment |
up vote
0
down vote
I would use something like:
#!/bin/bash
echo "What is the base name to reduce to unique lines? "
read filename
for i in *$filename*; do
cat $i >> tempfile
cat tempfile | sort -u -o tempfile
done
or
cat *filename* | sort -u
The placement of the *'s is based on how you name your itterations.
- When done read tempfile.
- Rename to whatever.
Use with caution in a test directory with test files first
- Tailor to your needs.
add a comment |
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
Pipe the result into sort to filter duplicates.
grep exampletext | sort -u
2
You are missing the files that you are going togrep.
– Nasir Riley
1 hour ago
3
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose ofsort -u...
– Jeff Schaller
42 mins ago
add a comment |
up vote
1
down vote
Pipe the result into sort to filter duplicates.
grep exampletext | sort -u
2
You are missing the files that you are going togrep.
– Nasir Riley
1 hour ago
3
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose ofsort -u...
– Jeff Schaller
42 mins ago
add a comment |
up vote
1
down vote
up vote
1
down vote
Pipe the result into sort to filter duplicates.
grep exampletext | sort -u
Pipe the result into sort to filter duplicates.
grep exampletext | sort -u
answered 1 hour ago
Jon Reinhold
828614
828614
2
You are missing the files that you are going togrep.
– Nasir Riley
1 hour ago
3
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose ofsort -u...
– Jeff Schaller
42 mins ago
add a comment |
2
You are missing the files that you are going togrep.
– Nasir Riley
1 hour ago
3
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose ofsort -u...
– Jeff Schaller
42 mins ago
2
2
You are missing the files that you are going to
grep.– Nasir Riley
1 hour ago
You are missing the files that you are going to
grep.– Nasir Riley
1 hour ago
3
3
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of
sort -u...– Jeff Schaller
42 mins ago
and then, once you supply multiple filenames, the filenames themselves would destroy the purpose of
sort -u...– Jeff Schaller
42 mins ago
add a comment |
up vote
0
down vote
If what you need is to find which file(s) match some text, use:
$ grep -rl 'text to find' ./dir
If you need only the first match of each file:
$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done
which will not print the name of the files matching, but will be fast.
Or:
$ find ../* -type f -exec sh -c '
a=$(sed -n "/echo/{p;q}" "$1");
[ "$a" ] && printf "%sn" "$1 : $a"
' findsh {} ;
If you need the filename also (separated with :).
add a comment |
up vote
0
down vote
If what you need is to find which file(s) match some text, use:
$ grep -rl 'text to find' ./dir
If you need only the first match of each file:
$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done
which will not print the name of the files matching, but will be fast.
Or:
$ find ../* -type f -exec sh -c '
a=$(sed -n "/echo/{p;q}" "$1");
[ "$a" ] && printf "%sn" "$1 : $a"
' findsh {} ;
If you need the filename also (separated with :).
add a comment |
up vote
0
down vote
up vote
0
down vote
If what you need is to find which file(s) match some text, use:
$ grep -rl 'text to find' ./dir
If you need only the first match of each file:
$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done
which will not print the name of the files matching, but will be fast.
Or:
$ find ../* -type f -exec sh -c '
a=$(sed -n "/echo/{p;q}" "$1");
[ "$a" ] && printf "%sn" "$1 : $a"
' findsh {} ;
If you need the filename also (separated with :).
If what you need is to find which file(s) match some text, use:
$ grep -rl 'text to find' ./dir
If you need only the first match of each file:
$ for file in ./*; do sed -n '/text to match/{p,q}' "$file"; done
which will not print the name of the files matching, but will be fast.
Or:
$ find ../* -type f -exec sh -c '
a=$(sed -n "/echo/{p;q}" "$1");
[ "$a" ] && printf "%sn" "$1 : $a"
' findsh {} ;
If you need the filename also (separated with :).
answered 52 mins ago
Isaac
9,70611445
9,70611445
add a comment |
add a comment |
up vote
0
down vote
I use:
grep test files* | puniq
puniq is: perl -ne '$seen{$_}++ or print;'
It is similar to sort -u but it does not sort the file.
1
This won't work for obvious reasons...
– don_crissti
31 mins ago
add a comment |
up vote
0
down vote
I use:
grep test files* | puniq
puniq is: perl -ne '$seen{$_}++ or print;'
It is similar to sort -u but it does not sort the file.
1
This won't work for obvious reasons...
– don_crissti
31 mins ago
add a comment |
up vote
0
down vote
up vote
0
down vote
I use:
grep test files* | puniq
puniq is: perl -ne '$seen{$_}++ or print;'
It is similar to sort -u but it does not sort the file.
I use:
grep test files* | puniq
puniq is: perl -ne '$seen{$_}++ or print;'
It is similar to sort -u but it does not sort the file.
answered 49 mins ago
Ole Tange
11.8k1448105
11.8k1448105
1
This won't work for obvious reasons...
– don_crissti
31 mins ago
add a comment |
1
This won't work for obvious reasons...
– don_crissti
31 mins ago
1
1
This won't work for obvious reasons...
– don_crissti
31 mins ago
This won't work for obvious reasons...
– don_crissti
31 mins ago
add a comment |
up vote
0
down vote
I would use something like:
#!/bin/bash
echo "What is the base name to reduce to unique lines? "
read filename
for i in *$filename*; do
cat $i >> tempfile
cat tempfile | sort -u -o tempfile
done
or
cat *filename* | sort -u
The placement of the *'s is based on how you name your itterations.
- When done read tempfile.
- Rename to whatever.
Use with caution in a test directory with test files first
- Tailor to your needs.
add a comment |
up vote
0
down vote
I would use something like:
#!/bin/bash
echo "What is the base name to reduce to unique lines? "
read filename
for i in *$filename*; do
cat $i >> tempfile
cat tempfile | sort -u -o tempfile
done
or
cat *filename* | sort -u
The placement of the *'s is based on how you name your itterations.
- When done read tempfile.
- Rename to whatever.
Use with caution in a test directory with test files first
- Tailor to your needs.
add a comment |
up vote
0
down vote
up vote
0
down vote
I would use something like:
#!/bin/bash
echo "What is the base name to reduce to unique lines? "
read filename
for i in *$filename*; do
cat $i >> tempfile
cat tempfile | sort -u -o tempfile
done
or
cat *filename* | sort -u
The placement of the *'s is based on how you name your itterations.
- When done read tempfile.
- Rename to whatever.
Use with caution in a test directory with test files first
- Tailor to your needs.
I would use something like:
#!/bin/bash
echo "What is the base name to reduce to unique lines? "
read filename
for i in *$filename*; do
cat $i >> tempfile
cat tempfile | sort -u -o tempfile
done
or
cat *filename* | sort -u
The placement of the *'s is based on how you name your itterations.
- When done read tempfile.
- Rename to whatever.
Use with caution in a test directory with test files first
- Tailor to your needs.
edited 39 mins ago
answered 1 hour ago
Michael Prokopec
47112
47112
add a comment |
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f484086%2fgrep-without-duplicates%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
What's your desired output ? Just the lines that match a certain pattern, without any filename prepended ? If so, you should be able to find the answer yourself...
– don_crissti
1 hour ago
1
An example would clarify the question.
– Kusalananda
57 mins ago
1
@Kusalananda - I wonder why none of the upvoters here takes the time to reply and enlighten us... This question "is clear" and "shows research effort" ?
– don_crissti
30 mins ago
@Don_crissti Desired output? Filename: Can but does not have to.
– neverMind9
21 mins ago
If it "does not have to" then, as I said, it should be simple but because your question is unclear people have different interpretations of the actual goal here...
– don_crissti
19 mins ago