Remove duplicate lines from data

[BillyConnite]

Hi all,

Every so often i like to remove duplicate lines from a list of raw data, if you want to do this with php, here's a simple way to do so:

<?php

/*
REMOVE DUPLICATE ENTRIES
DISPLAYS UNIQUE ENTIRES
*/

$list='random something
something or other
something
something else
something
randomness
chicken
bar snacks
something
foiled butter
O.o
...confuuussseeedd.........';

$listArray=explode("\n",$list);

$nonDupeList=array();

for($i=0;$i<count($listArray);$i++)
{
	if(!in_array($listArray[$i],$nonDupeList))
	{
	$nonDupeList[]=$listArray[$i];
	echo $listArray[$i]."<br />";
	}
}

?>

Hope you find it useful, i have countless times.

-Rhett.

 

[Eric]

A simpler way

<?php

$list = 'random something
something or other
something
something else
something
randomness
chicken
bar snacks
something
foiled butter
O.o
...confuuussseeedd.........';

$nonDupeList = array_unique(explode("\n", $list));

echo implode('<br />', $nonDupeList);

?>

 

[Kate]

If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host:

sort <your file> | uniq

There are plenty of things that can be done more efficiently than in PHP

 

[BillyConnite]

Thanks eric, i didn't realize there was an array_unique function, rep left .

Thanks sdsinc, i was specifically referring to php

 

[carob]

If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host:

sort <your file> | uniq

There are plenty of things that can be done more efficiently than in PHP

Just wondering, are we talking about rewriting the file or just formatting for output (similar to select distinct in SQL)?

 

[partiaran]

Sorry for the late reply. I use ddnr for duplicate text line removal. I have found it to be faster than php.

 

[structuresettlement]

Hi Billy Connite. I have tried this code but it didn't work.

 

[bizworldusa9]

in this first identified the duplicate lines are there from data.Identifies the duplicate lines,next step to remove the duplicate lines from data.




regards,
Bizworldusa.

 

[martiinko]

I usually do it like this:

// do a unique to check for dupes and stuff
// do the filter to filter out any possible empty items
$array = array_filter( array_unique( $array ), "strlen" );