Remove duplicate lines from data

BillyConnite · 04-11-2011, 05:17 AM

Hi all,

Every so often i like to remove duplicate lines from a list of raw data, if you want to do this with php, here's a simple way to do so:

PHP Code:

  <?php

 
/*

REMOVE DUPLICATE ENTRIES

DISPLAYS UNIQUE ENTIRES

*/

 
$list='random something

????: NamePros.com http://www.namepros.com/code/710519-remove-duplicate-lines-from-data.html
something or other

something

something else

something

randomness

chicken

bar snacks

something

foiled butter

O.o

...confuuussseeedd.........';

 
$listArray=explode("\n",$list);

 
$nonDupeList=array();

 
for($i=0;$i<count($listArray);$i++)

{

    if(!in_array($listArray[$i],$nonDupeList))

    {

    $nonDupeList[]=$listArray[$i];

    echo $listArray[$i]."<br />";

    }

}

 
?>

Hope you find it useful, i have countless times.

-Rhett.

**Eric** · 04-11-2011, 09:29 AM

A simpler way

????: NamePros.com http://www.namepros.com/showthread.php?t=710519

PHP Code:

  <?php

 
$list = 'random something

something or other

something

something else

something

randomness

chicken

bar snacks

something

foiled butter

O.o

...confuuussseeedd.........';

 
$nonDupeList = array_unique(explode("\n", $list));

 
echo implode('<br />', $nonDupeList);

 
?>

sdsinc · 04-11-2011, 12:50 PM

If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host:

Code:

sort <your file> | uniq

There are plenty of things that can be done more efficiently than in PHP

BillyConnite · 04-11-2011, 09:21 PM

Thanks eric, i didn't realize there was an array_unique function, rep left

.

Thanks sdsinc, i was specifically referring to php

carob · 04-11-2011, 10:22 PM

Originally Posted by sdsinc

If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host:

Code:

sort <your file> | uniq

There are plenty of things that can be done more efficiently than in PHP

Just wondering, are we talking about rewriting the file or just formatting for output (similar to select distinct in SQL)?

partiaran · 07-05-2011, 12:01 AM

Sorry for the late reply. I use ddnr for duplicate text line removal. I have found it to be faster than php.

structuresettlement · 03-02-2012, 12:18 AM

Hi Billy Connite. I have tried this code but it didn't work.

04-11-2011, 05:17 AM	THREAD STARTER #1 (permalink)
BillyConnite Join Date: Jul 2005 Location: Coffs H, Australia Posts: 3,456	Remove duplicate lines from data Hi all, Every so often i like to remove duplicate lines from a list of raw data, if you want to do this with php, here's a simple way to do so: PHP Code: <?php /* REMOVE DUPLICATE ENTRIES DISPLAYS UNIQUE ENTIRES / $list='random something ????: NamePros.com http://www.namepros.com/code/710519-remove-duplicate-lines-from-data.html something or other something something else something randomness chicken bar snacks something foiled butter O.o ...confuuussseeedd.........'; $listArray=explode("\n",$list); $nonDupeList=array(); for($i=0;$i<count($listArray);$i++) { if(!in_array($listArray[$i],$nonDupeList)) { $nonDupeList[]=$listArray[$i]; echo $listArray[$i]."<br />"; } } ?> Hope you find it useful, i have countless times. -Rhett. __________________ Free Forums / GoDaddy Coupon Codes (NEW DOMAIN!) / Free Arcade Script / <?='Your computer is '.(1?fine:broken).'.'?>*

04-11-2011, 09:29 AM	#2 (permalink)
Eric Tech Support Join Date: Mar 2005 Posts: 4,944	A simpler way ????: NamePros.com http://www.namepros.com/showthread.php?t=710519 PHP Code: `<?php $list = 'random something something or other something something else something randomness chicken bar snacks something foiled butter O.o ...confuuussseeedd.........'; $nonDupeList = array_unique(explode("\n", $list)); echo implode('<br />', $nonDupeList); ?>`

04-11-2011, 12:50 PM	#3 (permalink)
sdsinc Domains my Dominion Join Date: Aug 2005 Location: Web 1.0 Posts: 9,955	If you have a file and you want to remove the dupes, you could invoke the linux command line through the shell_exec function if allowed on your host: Code: sort <your file> \| uniq There are plenty of things that can be done more efficiently than in PHP __________________ NameNewsletter.com - free lists of available domain names ZoneFiles.net (beta) - ccTLD and gTLD droplists

04-11-2011, 09:21 PM	THREAD STARTER #4 (permalink)
BillyConnite Join Date: Jul 2005 Location: Coffs H, Australia Posts: 3,456	Thanks eric, i didn't realize there was an array_unique function, rep left . Thanks sdsinc, i was specifically referring to php __________________ Free Forums / GoDaddy Coupon Codes (NEW DOMAIN!) / Free Arcade Script / <?='Your computer is '.(1?fine:broken).'.'?> Last edited by BillyConnite; 04-11-2011 at 09:26 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Help Please - Need Script to Find & Remove Duplicate Links from a web page.	senaia	Programming	5	08-07-2006 01:55 PM
Dedicated Servers from just £45 $90 + [UK] 1 TB Data Transfer P4 3.0GHZ Only £119	blackicehosting	Web Hosting Offers	0	07-12-2005 06:24 PM
Unmetered Summer Savings + 10% Discount	blackicehosting	Web Hosting Offers	0	07-02-2005 11:20 AM
New Site New Looks New Unmetered Savings	blackicehosting	Web Hosting Offers	0	05-29-2005 02:58 PM
#1 Bring Power to your Hosting :: UK Redbus + LA Dedicated Servers	blackicehosting	Web Hosting Offers	0	05-11-2005 10:57 AM

07-05-2011, 12:01 AM	#6 (permalink)
partiaran First Time Poster! Join Date: Jul 2011 Posts: 1	Sorry for the late reply. I use ddnr for duplicate text line removal. I have found it to be faster than php.

03-02-2012, 12:18 AM	#7 (permalink)
structuresettlement New Member Join Date: Feb 2012 Location: Research,Listening rock music Posts: 2	Hi Billy Connite. I have tried this code but it didn't work.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)