Dynadot Aftermarket: discover domain gems worth investing in. Visit our aftermarket.

Remove duplicate lines from data

SpaceshipSpaceship
Namecheap AuctionsNamecheap Auctions
Namecheap AuctionsNamecheap Auctions
Watch
Hi all,

Every so often i like to remove duplicate lines from a list of raw data, if you want to do this with php, here's a simple way to do so:
PHP:
<?php

/*
REMOVE DUPLICATE ENTRIES
DISPLAYS UNIQUE ENTIRES
*/

$list='random something
something or other
something
something else
something
randomness
chicken
bar snacks
something
foiled butter
O.o
...confuuussseeedd.........';

$listArray=explode("\n",$list);

$nonDupeList=array();

for($i=0;$i<count($listArray);$i++)
{
	if(!in_array($listArray[$i],$nonDupeList))
	{
	$nonDupeList[]=$listArray[$i];
	echo $listArray[$i]."<br />";
	}
}

?>

Hope you find it useful, i have countless times.

-Rhett.
 
1
Close
The views expressed on this page by users and staff are their own, not those of NamePros.
GoDaddyGoDaddy
A simpler way :)

PHP:
<?php

$list = 'random something
something or other
something
something else
something
randomness
chicken
bar snacks
something
foiled butter
O.o
...confuuussseeedd.........';

$nonDupeList = array_unique(explode("\n", $list));

echo implode('<br />', $nonDupeList);

?>
 
2
Close
If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host:
Code:
sort <your file> | uniq
There are plenty of things that can be done more efficiently than in PHP :)
 
0
Close
Thanks eric, i didn't realize there was an array_unique function, rep left :).

Thanks sdsinc, i was specifically referring to php ;)
 
Last edited:
0
Close
If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host:
Code:
sort <your file> | uniq
There are plenty of things that can be done more efficiently than in PHP :)

Just wondering, are we talking about rewriting the file or just formatting for output (similar to select distinct in SQL)?
 
0
Close
Sorry for the late reply. I use ddnr for duplicate text line removal. I have found it to be faster than php.
 
0
Close
Hi Billy Connite. I have tried this code but it didn't work.
 
0
Close
in this first identified the duplicate lines are there from data.Identifies the duplicate lines,next step to remove the duplicate lines from data.




regards,
Bizworldusa.
 
0
Close
I usually do it like this:

PHP:
// do a unique to check for dupes and stuff
// do the filter to filter out any possible empty items
$array = array_filter( array_unique( $array ), "strlen" );
 
0
Close
DomainEasy: flexible LTO options. Close more domain deals with financing.

We're social

Escrow.com
Spaceship
Domain Recover
CryptoExchange.com

NamePros updates

Filters for activity feeds
Your news feed, Latest activity, and the activity tab on profiles now have filter buttons: All, Posts, Quick replies, and Other. Your...
AIVikings
Catchy
  • The sidebar remains visible by scrolling at a speed relative to the page’s height.
Back